Electronic device for tracking external object in video and its method

Abstract

To provide an electronic device for acquiring labeling information to be used for training of a model for recognizing an object from a video including a plurality of images and its method.SOLUTION: A method identifies a first position related to an external object in a first image of a plurality of images of a video and first timing of the plurality of images, identifies a second position related to the external object in a second image of second timing after the first timing in the plurality of images on the basis of the first position in the first image, acquires one or more third positions related to the external object, corresponding to one or more third images included in a time section between the first timing and the second timing on the basis of the first position and the second position, and stores the first position, the one or more third positions, and the second position as labeling information showing movement of the external object identified in the time section of the video.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an electronic device and method for tracking an external object in a video. [Background technology]

[0002] As image object recognition technology develops, various forms of services are emerging. These services can be used for Automatic Driving, Augmented Reality, Virtual Reality, Metaverse, etc., and can be provided through electronic devices owned by different users, such as smartphones. The services can be related to hardware and / or software mechanisms that mimic human behavior and / or thinking, such as artificial intelligence. Technologies related to artificial intelligence can include technologies that utilize neural networks that simulate the neural networks of living organisms. Summary of the Invention [Problem to be solved by the invention]

[0003] For video playback, methods are being investigated to more quickly identify in images (eg, frames) contained in the video the location of external objects commonly captured in the images. [Means for solving the problem]

[0004] According to an embodiment, an electronic device may include a memory and a processor. The processor may be configured to identify, from the memory, a first position associated with an external object in a plurality of images for a video and a first image of the plurality of images at a first timing. The processor may be configured to identify, based on the first position in the first image, a second position associated with the external object in a second image of the plurality of images at a second timing after the first timing. The processor may be configured to obtain, based on the first position and the second position, one or more third positions associated with the external object corresponding to one or more third images included in a time interval between the first timing and the second timing. The processor may be configured to store the first position, the one or more third positions, and the second position as labeling information indicating a movement of the external object identified in the time interval of the video.

[0005] According to one embodiment, a method of an electronic device may include identifying, from a memory of the electronic device, a first location associated with an external object within a plurality of images for a video and a first image of the plurality of images at a first time. The method may include identifying, at the first location within the first image, a second location associated with the external object within a second image of the plurality of images at a second time after the first time. The method may include obtaining, based on the first location and the second location, one or more third locations associated with the external object corresponding to one or more third images included in a time interval between the first time and the second time. The method may include storing the first location, the one or more third locations, and the second location as labeling information indicative of a movement of the external object identified within the time interval of the video.

[0006] According to one embodiment, an electronic device may include a display, a memory, and a processor. The processor may be configured to identify a first input indicating a selection of a first position associated with an external object within a first image of a video stored in the memory while displaying the first image on the display. The processor may be configured to perform a first type of calculation for recognizing the external object based on the first input to identify a second position associated with the external object within a second image of a plurality of images for the video following a time interval starting from a timing of the first image. The processor may be configured to perform a second type of calculation for interpolating the first position and the second position to obtain a third position associated with the external object within one or more third images included in the interval. The processor may be configured to display any of the first image, the one or more third images, and the second image on the display in response to a second input indicating playback of at least a portion of the video included in the time interval, and to display a visual object corresponding to any of the first position, the third position, or the second position, superimposed on the image displayed on the display.

[0007] According to one embodiment, a method of an electronic device may include, while displaying a first image of a video stored in a memory of the electronic device on a display of the electronic device, identifying a first input indicating a selection of a first position associated with an external object in the first image. The method may include, based on the first input, performing a first type of calculation for recognizing the external object to identify a second position associated with the external object in a second image of a plurality of images for the video following a time interval starting from a timing of the first image. The method may include performing a second type of calculation for interpolating the first position and the second position to obtain a third position associated with the external object in one or more third images included in the time interval. The method may include, in response to a second input indicating playback of at least a portion of the video included in the time interval, displaying on the display any of the first image, the one or more third images, and the second image, and displaying a visual object corresponding to any of the first position, the third position, or the second position, superimposed on the image displayed on the display. Effect of the Invention

[0008] According to one embodiment, an electronic device can obtain labeling information used to train a model to recognize objects from a video that includes multiple images recorded sequentially.

[0009] According to one embodiment, an electronic device can more quickly obtain labeling information including the locations of visual objects included in each of multiple images included in a video. [Brief description of the drawings]

[0010] [Figure 1a] 1 illustrates an example of an operation for obtaining labeling information by an electronic device according to one embodiment. [Figure 1b] 1 illustrates an example of an operation for obtaining labeling information by an electronic device according to one embodiment. [Diagram 2] 1 illustrates an example block diagram of an electronic device, according to one embodiment. [Diagram 3] 1 illustrates an example flowchart of operations performed by an electronic device, according to one embodiment. [Figure 4] 1 illustrates an example of an operation of recognizing an object from an image contained in a video by an electronic device according to one embodiment. [Figure 5a] 1 shows an example of the operation of an electronic device that obtains labeling information of another image based on labeling information of two images included in a video. [Figure 5b] 1 shows an example of the operation of an electronic device that obtains labeling information of another image based on labeling information of two images included in a video. [Figure 6] 1 illustrates an example flowchart of operations performed by an electronic device, according to one embodiment. [Figure 7] 1 illustrates an example of an interpolation performed by an electronic device, according to one embodiment. [Figure 8] FIG. 1 illustrates an example of a block diagram illustrating an autonomous driving system for a vehicle, according to one embodiment. [Figure 9] FIG. 1 illustrates an example of a block diagram illustrating an autonomous vehicle, according to one embodiment. [Figure 10] FIG. 1 illustrates an example of a block diagram illustrating an autonomous vehicle, according to one embodiment. [Figure 11] 1 illustrates an example of a gateway associated with a user device, according to one embodiment. [Figure 12] FIG. 1 illustrates the operation of an electronic device for training a neural network based on a set of training data. [Figure 13] FIG. 1 is a block diagram of an electronic device, according to one embodiment. [Figure 14] FIG. 1 is a block diagram of an electronic device, according to one embodiment. [Figure 15]1 is a diagram illustrating a process of obtaining labeling data related to a region selected by a user input in an image according to an embodiment of the present invention; [Figure 16] 1 is a diagram illustrating a process of generating labeling data for learning to identify an object in an image according to an embodiment of the present invention. [Figure 17] FIG. 2 is a diagram for explaining a hierarchical structure of labeling data generated according to an embodiment of the present invention. [Figure 18] 10 is a diagram for explaining a hierarchical structure of information regarding external objects in labeling data generated by an embodiment of the present invention. FIG.

[0011] [Figure 19] FIG. 11 is a diagram showing an example of coordinates assigned to Points parameters when the Shape Type is "Polygon" according to one embodiment of the present invention.

[0012] [Figure 20] FIG. 13 is a diagram illustrating an example of dividing labeling data generated for efficient training of a neural network according to an embodiment of the present invention into a training set, a validation set, and / or a test set. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Various embodiments of the present document will now be described with reference to the accompanying drawings.

[0014] It should be understood that the various embodiments and the terms used therein are not intended to limit the technology described in this document to a particular embodiment, but include various modifications, equivalents, or alternatives of the embodiment. With respect to the description of the drawings, similar components may use similar reference numerals. Unless the context clearly dictates otherwise, singular expressions may include plural expressions. In this document, expressions such as "A or B," "at least one of A and / or B," "A, B or C," or "at least one of A, B, and C" may include all possible combinations of the items listed together. Expressions such as "first," "second," or "first" or "second" may modify the corresponding components regardless of order or importance, and are used only to distinguish a certain component from other components, not to limit the corresponding components. When a (e.g., first) component is referred to as being "coupled (functionally or communicatively)" or "connected" to another (e.g., second) component, the component may be directly connected to the other component or may be connected through another component (e.g., a third component).

[0015] The term "module" as used in this document includes a unit configured in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrated component or the smallest unit or portion of a component that performs one or more functions. For example, a module may be configured in an application-specific integrated circuit (ASIC).

[0016] 1a-1b show an example of an operation of acquiring labeling information by an electronic device 101 according to an embodiment. Referring to FIG. 1a-1b, form factors of the electronic device 101 are exemplarily shown. From the viewpoint of a user's ownership, the electronic device 101 may be referred to as a terminal, a user device, and / or a user terminal. The electronic device 101 may include a personal computer (PC), such as a desktop 101-1 and / or a laptop 101-2. The electronic device 101 may include a smartphone 101-3, a smartpad, and / or a tablet PC. The terminal may include a smart accessory, such as a smartwatch and / or a head-mounted device (HMD).

[0017] Referring to FIG. 1a, an electronic device 101 according to an embodiment can display a screen 110. The electronic device 101 can visualize information (e.g., a video 120) using the screen 110. One or more hardware components included in the electronic device 101 for displaying the screen 110 and / or outputting information are exemplarily described with reference to FIG. 2. Hereinafter, the screen 110 may refer to a user interface (UI) displayed in at least a portion of a display. The screen may include, for example, a window in a Windows operating system and / or an activity in an Android operating system. The embodiment is not limited thereto, and the screen 110 may be formed in an external space by light output from the electronic device 101 to the external space. For example, the screen 110 may be formed on a plane on which light is projected in the external space. For example, the screen 110 may be displayed three-dimensionally in the form of a hologram in the external space.

[0018] According to one embodiment, the electronic device 101 can display a screen 110 for playing a video 120. The video 120 can include a set of images that can be displayed sequentially according to a frame rate (or frames per second (fps)). The images included in the set may be referred to as frames, frame data, and / or frame images. The video 120 can include an audio signal that is output while the images are displayed sequentially. In terms of including both visual information (e.g., a set of images) and auditory information (e.g., an audio signal), the video 120 may be referred to as a multimedia content (or media content).

[0019] Referring to FIG. 1a, a plurality of images 120-1, 120-k, 120-N identified by the electronic device 101 from the video 120 are exemplarily shown. Assume that in a state where the video 120 is being played, the electronic device 101 displays an image (e.g., a first image 120-1) shown on the left side of FIG. 1a before another image (e.g., a k-th image 120-k). For example, in the state, the k-th image 120-k may be displayed by the electronic device 101 after the first image 120-1. For example, in the state, the N-th image 120-N may be displayed by the electronic device 101 after the k-th image 120-k. For example, the electronic device 101 may sequentially display the first image 120-1, the k-th image 120-k, and the N-th image 120-N in this order.

[0020] According to an embodiment, the electronic device 101 can recognize one or more external objects from the video 120. According to an embodiment, the external objects can include pedestrians, vehicles, bikes, personal mobility (PM) road signs, lane markings, etc., as objects present in the surroundings of a system (e.g., a vehicle) to which the electronic device 101 is attached. Recognizing the one or more external objects can include generating and / or obtaining information from the video 120 about the one or more external objects captured by the video 120. The information can include data indicative of portions of at least one of a plurality of images for the video 120 associated with the one or more external objects. The information can include data indicative of a classification or category of the one or more external objects. In an embodiment, recognizing the one or more external objects by the electronic device 101 can include generating and / or obtaining information indicative of a location associated with the one or more external objects in at least one of a plurality of images included in the video 120. According to one embodiment, electronic device 101 may obtain information including results of recognizing one or more external objects based on recognizing one or more external objects from video 120. The information obtained by electronic device 101 may be referred to as metadata and / or labeling information corresponding to video 120.

[0021] According to an embodiment, information about one or more external objects acquired by the electronic device 101 and associated with the video 120 may be referred to as labeling information in terms of training a model for recognizing the external objects. For example, the labeling information, together with the video 120 that matches the labeling information, may be used for supervised learning of a model. The model is a recognition model implemented in software or hardware that uses a large number of artificial neurons (or nodes, perceptrons) to mimic the computational capabilities of biological systems. Based on the model, the electronic device 101 may perform operations similar to human cognitive functions and learning processes. The supervised learning of a model may include an operation of modifying weights assigned to multiple nodes and / or connections between multiple nodes included in the model using input data (e.g., the video 120), the model's output data of the input data, and ground truth data (e.g., the labeling information corresponding to the video 120). A model trained with video 120 and labeling information corresponding to video 120 can output a result of recognizing an external object from another video other than video 120 when the other video is received. Information about the one or more external objects can be used for purposes other than supervised learning. For example, from the perspective of tracking and / or monitoring an external object contained in video 120, information about one or more external objects associated with video 120 may be referred to as object tracking information.

[0022] 1a, a screen 110 displayed by an electronic device 101 to generate labeling information corresponding to a video 120 is exemplarily shown. The electronic device 101 can display a first image 120-1 of a first timing of the video 120. The electronic device 101 can display a panel 112 including one or more visual objects for controlling the video 120 together with the first image 120-1 in the screen 110. The visual object can mean an object that can be deployed in the screen 110 for information transmission and / or interaction, such as a text, an image, an icon, a video, a button, a check box, a radio button, a text box, a slider, and / or a table. The visual object may be referred to as a visual element, a UI element, a view object, and / or a view element. While displaying the first image 120-1 in the video 120, the electronic device 101 can identify a user input for selecting a region in the first image 120-1 that is associated with an external object. An exemplary operation of the electronic device 101 for identifying a user input is described with reference to FIG.

[0023] 1a, an electronic device 101 displaying a first image 120-1 including a visual object 130-1 representing a vehicle in the first image 120-1 may identify a user input for selecting and / or segmenting the visual object 130-1. For example, the electronic device 101 may identify an input indicating a selection of apexes A1, B1, C1, D1 of a region 140-1 of the first image 120-1 including the visual object 130-1.

[0024] In one embodiment, in response to an input indicating a selection of vertices A1, B1, C1, D1, the electronic device 101 can display a visual object in a shape of a bounding box representing the region 140-1 corresponding to the input. For example, the vertices of the bounding box can match the vertices A1, B1, C1, D1 of the region 140-1. The electronic device 101 can display the bounding box by connecting the vertices A1, B1, C1, D1 with lines having a designated color. The designated color of the line for displaying the bounding box can be related to the result of recognizing the visual object 130-1 contained in the region 140-1. In the exemplary case of FIG. 1a, based on identifying that the external object corresponding to the visual object 130-1 is a vehicle, the electronic device 101 can set the designated color of the bounding box as a designated color corresponding to a vehicle. Embodiments are not limited in this respect, and the electronic device 101 may display the results of recognizing an external object in the region 140-1 of the first image 120-1 contained within the bounding box by displaying text (e.g., "Vehicle") adjacent to or superimposed on the bounding box.

[0025] According to one embodiment, the electronic device 101 can identify one or more locations associated with the external object from other images different from the first image 120-1 (e.g., the kth image 120-k and / or the Nth image 120-N) based on identifying a location associated with the external object in the first image 120-1 (e.g., the region 140-1). In one embodiment, the electronic device 101 can identify a location associated with the external object from other images different from the first image 120-1 based on a user input to select a location in the first image 120-1 associated with the external object (e.g., the location of the region 140-1). The electronic device 101 can identify a location associated with the external object from other images different from the first image 120-1 without requiring any other user input distinct from the user input corresponding to the first image 120-1. Operation of the electronic device 101 to identify a location of an external object in other images in the video 120 based on a user input to the first image 120-1 will be described with reference to FIG. 3 and / or FIG. 6.

[0026] According to one embodiment, the electronic device 101 can identify a second location associated with an external object (in the exemplary case of FIG. 1a, a vehicle) associated with the region 140-1 in a second image at a second time in the video 120 after the first time in the first image 120-1 based on the region 140-1 identified by the input associated with the first image 120-1. To identify the second location, the electronic device 101 can compare feature information (e.g., one or more feature points) of the first image 120-1 and the second image, respectively. To identify the second location, the electronic device 101 can use a model trained to recognize objects. The operation of the electronic device 101 to identify the second location based on the feature information will be described with reference to FIG. 4.

[0027] In a non-limiting example, extracting feature information of all of the other images (image frames) included in the video 120 that are different from the first image 120-1 or inputting all of the other images into the model may cause an increase in the amount of calculation and / or power consumption. This may impose a lot of load on the electronic device 101 according to an embodiment of the present invention, and therefore may have a negative impact on the efficient operation of a system for driving assistance of a driver of a vehicle to be driven in a road environment, particularly an urban environment, where highly variable and unpredictable situations frequently occur. According to an embodiment, the electronic device 101 may change a computing method for recognizing an external object while recognizing an external object from the other images based on the region 140-1 of the first image 120-1 selected by the user. The electronic device 101 may change the computing method applied to each of the other images among the different computing methods based on the timing of each of the other images in the video 120. The computational methods may include a first type of computational method for recognizing an external object based on feature points of an image, and a second type of computational method related to the interpolation of positions of certain external objects identified from different images.

[0028] Referring to FIG. 1a, based on an input indicating a location (e.g., region 140-1) in the first image 120-1 that is associated with an external object, the electronic device 101 may select an Nth (N th ) image (or Nth image (N thIn the Nth image 120-N, which is an image located at a timing of (a) the first image 120-1 and (b) the second image 120-2, a position (e.g., a region 140-N) associated with an external object can be identified. For example, the electronic device 101 can compare one or more feature points included in the region 140-1 of the first image 120-1 with feature points of the Nth image 120-N to identify the region 140-N associated with an external object in the Nth image 120-N. When displaying the Nth image 120-N on the screen 110, the electronic device 101 can display a visual object having a form of a bounding box corresponding to the region 140-N, superimposed on the Nth image 120-N. For example, the electronic device 101 can display a bounding box having a form of a line connecting the vertices AN, BN, CN, and DN of the region 140-N.

[0029] Referring to Fig. 1a, upon identifying the region 140-N of the Nth image 120-N based on the feature points in the region 140-1 of the first image 120-1, the electronic device 101 can identify locations associated with external objects corresponding to the regions 140-1, 140-N in one or more images included in the time interval of the video 120 bounded by the first image 120-1 and the Nth image 120-N. Referring to Fig. 1a, the time interval of the video 120 bounded by the first image 120-1 and the Nth image 120-N may have a beginning timing of the first image 120-1 or an end timing of the second timing of the Nth image 120-N. In one embodiment, the electronic device 101 can identify a region 140-k associated with the kth image 120-k (where "k" is a natural number greater than 1 and less than N) included in the time interval based on interpolation to a region 140-1 of the first image 120-1 and a region 140-N of the Nth image 120-N.

[0030] In one embodiment, within identifying region 140-k based on the interpolation of regions 140-1, 140-N, electronic device 101 can determine the coordinates of vertices Ak, Bk, Ck, Dk of region 140-k in kth image 120-k based on the coordinates of vertices A1, B1, C1, D1 of region 120-1 and the coordinates of vertices AN, BN, CN, DN of region 140-k. For example, the coordinate of vertex Ak of ​​region 140-k may be the coordinate of an internally dividing point of a line connecting vertex A1 of region 140-1 and vertex AN of region 140-N. The proportion by which the line is divided by this internally dividing point may be related to the timing that kth image 120-k has within the time interval. An example of the operation of the electronic device 101 for identifying the region 140-k of the kth image 120-k and / or the vertices of the region 140-k based on the interpolation of the vertices of the regions 140-1, 140-N will be described with reference to Figures 5a-5b.

[0031] With reference to FIG. 1a, the operation of the electronic device 101 for recognizing the regions 140-1, 140-k, 140-N associated with an external object in the first image 120-1, the kth image 120-k, and the Nth image 120-N has been described, but the embodiment is not limited thereto. For example, the electronic device 101 may identify the region associated with the external object based on the interpolation of the regions 140-1, 140-N for all images between the first image and the Nth image, including the kth image. Although not shown, for other images after the Nth image in the video 120, the electronic device 101 may identify the region associated with the external object based on the feature points and / or the interpolation. According to an embodiment, the electronic device 101 may store the result of identifying the region associated with the external object from the multiple images included in the video 120 as labeling information corresponding to the video 120.

[0032] According to one embodiment, the electronic device 101 can visualize in the screen 110 the identification of regions associated with an external object from each of a plurality of images for the video 120. The electronic device 101 can identify inputs from the user for adjusting (or calibrating) the identification of said regions, as described below with reference to FIG.

[0033] 1b, an exemplary operation of the electronic device 101 for identifying an input indicating a selection of an external object in a particular image (e.g., the first image 120-1) of the video 120 is described. Referring to FIG. 1b, the electronic device 101 according to an embodiment can display a screen (e.g., the screen 150-1, 150-2) for obtaining labeling data corresponding to the first image 120-1. The electronic device 101 can display one of the screens 150-1, 150-2 of FIG. 1b based on an input indicating a selection of the first image 120-1 among the multiple images included in the video 120.

[0034] Referring to FIG. 1b, based on the screen 150-1 corresponding to the first image 120-1 of the video 120, the electronic device 101 can identify an input indicating a selection of the rectangular region 160-1. The input indicating the selection of the region 160-1 can include a drag input from vertex A1 to vertex A3 of the region 160-1, identified based on a mouse connected to the electronic device 101. The embodiment is not limited thereto, and the electronic device 101 can identify the input indicating the selection of the region 160-1 based on a drag gesture made on a touch screen. Having identified the input, the electronic device 101 can obtain a class (or category) of the object captured through the region 160-1. For example, the electronic device 101 can display a pop-up window for receiving text representing the class. Based on the text received from the user through the pop-up window, the electronic device 101 can identify the class (or category) of the object (e.g., visual object 130-1) captured in the region 160-1. The electronic device 101 may store information about the region 160-1 (eg, the coordinates, classes and / or categories of the vertices A1, A2, A3, A4) as labeling data corresponding to the first image 120-1.

[0035] In one embodiment, the shape of the region formed in the first image 120-1 to obtain the labeling data is not limited to a rectangle (e.g., the region 160-1 in FIG. 1b). With reference to the screen 150-2 in FIG. 1b, an input indicating the selection of the polygonal region 160-2 from the first image 120-1 of the video 120 can be identified. The input indicating the selection of the region 160-2 can be identified by a gesture of sequentially clicking and / or touching the vertices B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, and B14 of the polygon. Having identified the input, the electronic device 101 can display the polygonal region 160-2 formed in the first image 120-1 by connecting the vertices, as in the screen 150-2 in FIG. 1b. The electronic device 101 can identify an input indicating a selection of the region 160-2 as well as an input for setting a class (or category) of the visual object 130-1 in the first image 120-1 partitioned by the region 160-2. The electronic device 101 can store the coordinates of the vertices B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14 of the region 160-2 and the combination of the classes (or categories) as labeling data of the first image 120-1.

[0036] As described above, according to an embodiment, the electronic device 101 may recognize an external object in another image that is distinct from a specific image (e.g., a first image 120-1) among a plurality of images included in the video 120 based on an input indicating a selection of the external object in the specific image. The electronic device 101 may recognize the external object based on a combination of a first type of calculation method that requires a relatively large amount of calculations based on feature points and / or models and a second type of calculation method that requires a relatively small amount of calculations based on interpolation. For example, the electronic device 101 may perform an operation of recognizing an external object across a plurality of images in the video 120 without recognizing the external object in other images different from the specific image. Since the electronic device 101 alternately applies the calculation methods that require different amounts of calculations to different images in the video 120, the amount of calculations required to obtain labeling information corresponding to the video 120 may be reduced. Since the amount of calculations required to obtain the labeling information is reduced, the electronic device 101 may obtain the labeling information corresponding to the video 120 more quickly.

[0037] In the following, an exemplary structure of electronic device 101 for performing the operations described with reference to FIGS. 1a-1b will be described with reference to FIG.

[0038] FIG. 2 illustrates an example of a block diagram of an electronic device 101 according to an embodiment. The electronic device 101 of FIG. 2 may include the electronic device 101 of FIG. 1a-FIG. 1b. The electronic device 101 of FIG. 2 may include at least one of a processor 210, a memory 220, or a display 230. The processor 210, the memory 220, and the display 230 may be electrically and / or operably coupled with each other by an electronic component such as a communication bus 202. Hereinafter, the term "operably coupled" may mean that a direct connection between the hardware or an indirect connection between the hardware is established by wire or wirelessly such that a first piece of hardware controls a second piece of hardware. Although illustrated based on different blocks, the embodiment is not limited thereto, and a part of the hardware of FIG. 2 (e.g., at least a part of the processor 210 and the memory 220) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware included in electronic device 101 is not limited to that shown in Figure 2. For example, electronic device 101 may include only a portion of the hardware components shown in Figure 2.

[0039] According to an embodiment, the processor 210 of the electronic device 101 may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The number of processors 210 may be one or more. For example, the processor 210 may have a multi-core processor structure, such as a dual core, a quad core, or a hexa core.

[0040] According to one embodiment, the memory 220 of the electronic device 101 may include hardware components for storing data and / or instructions input and / or output to the processor 210. The memory 220 may include, for example, a volatile memory, such as a random-access memory (RAM), and / or a non-volatile memory, such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM (RAM), and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disk, a solid state drive (SSD), and an embedded multimedia card (eMMC).

[0041] According to an embodiment, one or more instructions may be stored in the memory 220 of the electronic device 101, the instructions indicating operations and / or actions that the processor 210 performs on data. The set of one or more instructions may include a firmware, an operating system, a process, a routine, a subroutine, and / or an application. For example, the electronic device 101 and / or the processor 210 may perform at least one of the operations of FIG. 3 and / or FIG. 6 when a set of a plurality of instructions distributed in the form of an operating system, a firmware, a driver, and / or an application is executed. In the following, an application being installed on the electronic device 101 may mean that one or more instructions provided in the form of an application are stored in the memory 220 of the electronic device 101, and thus the one or more applications are stored in a format executable by the processor 210 of the electronic device 101 (e.g., a file having an extension specified by the operating system of the electronic device 101).

[0042] According to an embodiment, the display 230 of the electronic device 101 may output visualized information (e.g., the screen of FIG. 1a) to a user. For example, the display 230 may be controlled by a controller such as a graphic processing unit (GPU) and output visualized information to a user. The display 230 may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The display 230 may include a flat panel display (FPD) and / or electronic paper. The embodiment is not limited thereto, and the display 230 may have an at least partially curved form or a deformable form.

[0043] According to an embodiment, the electronic device 101 can obtain the labeling information 240 from the video 120 stored in the memory 220. The video 120 of FIG. 2 can include the video 120 of FIG. 1a-FIG. 1b. For example, the electronic device 101 can identify from the video 120 a plurality of images (e.g., the first image 120-1, the kth image 120-k, and / or the Nth image 120-N of FIG. 1a-FIG. 1b) encoded or compressed within the video 120 in order to play the video 120. The labeling information 240 obtained by the electronic device 101 from the video 120 can include the location of an external object associated with each of the plurality of images. According to an embodiment, the electronic device 101 can obtain the labeling information 240 for the entire plurality of images based on the location of an external object associated with the several images without user input (e.g., the input described with reference to FIG. 1b) for the entire plurality of images.

[0044] According to an embodiment, the electronic device 101 can interpolate positions corresponding to two images separated from each other in the sequence of images included in the video 120 to obtain positions corresponding to other images between the two images. The positions of the external objects associated with the two separated images can be identified by a model included in the electronic device 101 for the analysis of the user of the electronic device 101 and / or the feature points. If the length of the time interval between the two separated images is relatively long (e.g., exceeds a specified threshold length), the electronic device 101 can perform a calibration of the positions obtained by the interpolation based on the feature points of one or more images in the time interval. According to an embodiment, the labeling information 240 stored in the memory 220 by the electronic device 101 can include data indicating positions associated with the external objects in each image. The data can include at least one of the coordinates, width, height, aspect ratio, or size of at least one of the vertices of the regions 140-1, 140-k, 140-N of FIG. 1a.

[0045] According to an embodiment, the labeling information 240 acquired by the electronic device 101 can represent the movement of an external object captured in the video 120. For example, the labeling information 240 can include the location of an area associated with an external object in each of the images included in the video 120 within a time period in which the images are played sequentially. The electronic device 101 can identify a result of recognizing an external object in each of the images of the video 120 based on the labeling information 240. The result can include the location of an area associated with an external object in a particular image. The electronic device 101 can tune (or train) a model (or neural network) for recognizing an external object using the images and the labeling information 240. Tuning of the model based on a relatively large number of images may be required to improve the performance of the model. According to an embodiment, the electronic device 101 can generate the labeling information 240 for the entire images included in the video 120 using information about an external object associated with a particular image of the video 120 (e.g., information indicating an area associated with an external object in a particular image). Because the electronic device 101 generates labeling information 240 across multiple images, the electronic device 101 can more quickly acquire the number of images required to tune the model.

[0046] In the following, an example of operations performed by the electronic device 101 of FIG. 2 will be described with reference to FIG.

[0047] 3 illustrates an example of a flow chart of operations performed by an electronic device, according to one embodiment. The electronic device 101 and / or the processor 210 of FIGS. 1a-1b and 2 may perform the operations described with reference to FIG.

[0048] Referring to FIG. 3, in an embodiment, in operation 310, a processor of the electronic device can identify a plurality of images for a video. The video of operation 310 can include video 120 of FIGS. 1a-2. The processor can identify a plurality of images for playing the video from a file having a format related to the motion picture experts group (MPEG). The file can be stored in a memory (e.g., memory 220 of FIG. 2) or can be received on a streaming basis from a network (e.g., the Internet) connected to the electronic device. The processor can identify a plurality of images and an order in which the plurality of images are to be played based on operation 310.

[0049] 3, in operation 320, according to one embodiment, a processor of the electronic device may identify a first location associated with an external object in a first image of the plurality of images at a first timing. The processor may identify a first input (e.g., the input described with reference to FIG. 1b) indicating a selection of the first location while displaying the first image on a display (e.g., display 230 of FIG. 2). The first image of operation 320 may include the first image 120-1 of FIG. 1a. For example, the screen 110 of FIG. 1a may be displayed by the processor having identified an input indicating a selection of the region 140-1 of the first image 120-1. Based on the input indicating the selection of the region 140-1 of FIG. 1a, the processor may identify a first location associated with the external object (e.g., a location of the region 140-1) in the first image 140-1. In one embodiment, the first location identified by the processor based on operation 320 may include at least one coordinate associated with a visual object associated with the external object in the first image (e.g., visual object 130-1 of FIG. 1a). At least one coordinate may be contained within a region 140-1 bounded by vertices A1, B1, C1, D1 in image 120-1 of FIG. 1a.

[0050] 3, in an operation 330, according to an embodiment, a processor of the electronic device may identify a second location associated with an external object in a second image of the plurality of images at a second timing after the first timing based on the first location of the operation 320. Upon identifying a first input for identifying the first location of the operation 320, the processor may perform a first type of calculation for recognizing an external object based on the first input to identify a second location in the second image after a time interval starting from the first timing. In the sequence of the plurality of images for playing the video of the operation 310, one or more third images may be disposed between the first image of the first timing and the second image of the second timing.

[0051] In one embodiment, a first type of calculation performed by the processor to identify the second location of operation 330 may relate to feature points of the first image and the second image. For example, the processor may extract at least one feature point from the first image that is associated with the first location of the first image identified by operation 320. The processor may extract one or more feature points from the second image of operation 330. The processor may compare the at least one feature point extracted from the first image with one or more feature points extracted from the second image. Based on this comparison, the processor may identify a region of the second image (e.g., region 140-N of FIG. 1a) that is similar to a region that includes the first location in the first image (e.g., region 140-1 of FIG. 1a). The second location of operation 330 may be included within the region of the second image identified by the processor.

[0052] Embodiments are not limited in this respect, and the first type of calculation performed by the processor to identify the second location of operation 330 may be associated with a model trained to recognize external objects. For example, the processor may input data regarding the external object associated with the first location in the first image, along with the second image, into the model. Based on the data and data output from the model to which the second image was input, the processor may identify a second location associated with the external object in the second image.

[0053] Referring to FIG. 3, in operation 340, according to an embodiment, the processor of the electronic device may obtain one or more third positions corresponding to one or more third images included in the time interval between the first timing and the second timing and related to the external object based on the first position and the second position. For example, the processor may perform a second type of calculation to interpolate the first position of operation 320 and the second position of operation 330 to obtain one or more third positions. The one or more third positions obtained by the processor may be located on a line connecting the first position and the second position. The one or more third positions on the line may be associated with the timing of one or more third images. The processor may obtain the one or more third positions of operation 340 based on a calculation amount less than that performed by operation 330 to extract feature points.

[0054] Referring to FIG. 3, in operation 350, according to one embodiment, a processor of the electronic device may store the first location, one or more third locations, and the second location as labeling information indicative of the movement of the identified external object within the time interval of the video. The labeling information 240 of FIG. 2 may include the labeling information of operation 350. According to one embodiment, the processor may store in the labeling information parameters representative of the image of the plurality of images corresponding to the location identified by operation 320 to operation 340. For example, the labeling information stored by the processor may include a parameter indicating that the first location corresponds to the first image (e.g., a timestamp linked to the data indicative of the first location). For example, the labeling information stored by the processor may include a parameter representing the mapping of the second location and the second image. For example, the labeling information stored by the processor may include a parameter representing the relationship of the one or more third locations and the one or more third images.

[0055] As mentioned above, according to an embodiment, the processor of the electronic device can perform different types of calculations for obtaining labeling information for a plurality of images included in the video. The labeling information obtained by the processor can be used to train a model based on the plurality of images. According to an embodiment, the processor of the electronic device can display a screen for receiving a user's feedback on the labeling information to improve the accuracy of the labeling information. For example, the processor can display any of the first image, one or more third images, and the second image on a display (e.g., display 230 of FIG. 2). The processor can display any of the first image, one or more third images, and the second image on the display in response to a second input indicating playback of at least a portion of the video included in the time interval of operation 340. The processor can display a visual object superimposed on the image displayed on the display and corresponding to any of the first position, the third position, or the second position included in the labeling information. The visual object can take the form of a bounding box. Based on the visual object, the processor of the electronic device can receive an input for at least partially adjusting the labeling information. The input for at least partially adjusting the labeling information may be received similar to the input indicating a selection of a region (eg, regions 160-1, 160-2) described with reference to FIG. 1b.

[0056] An example of the operations 330, 340 of FIG. 3 performed by an electronic device according to one embodiment will now be described with reference to FIGS. 4 and 5a-5b.

[0057] 4 illustrates an example of an operation of recognizing an object from an image included in a video by an electronic device, according to one embodiment. The operation of the electronic device described with reference to FIG. 4 may be performed by the electronic device 101 and / or the processor 210 of FIGS. 1a-1b and 2. The operation of the electronic device described with reference to FIG. 4 may be related to at least one of the operations of FIG. 3 (e.g., operation 330).

[0058] Referring to FIG. 4, according to one embodiment, the electronic device can identify an external object (e.g., a vehicle) commonly associated with a first image 120-1 and an Nth image 120-N spaced apart from one another in a sequence of images included in a video 120. The electronic device can identify the first image 120-1 and a visual object 130-1 corresponding to the external object in the first image 120-1 based on operation 320 of FIG. 3. In one embodiment, the electronic device can identify a position of the visual object 130-1 associated with the external object in the first image 120-1 based on an input indicating a selection of the region 140-1. The input indicating a selection of the region 140-1 may include the input of FIG. 1b. The position of the visual object 130-1 may correspond to the first position of FIG. 3.

[0059] 4, according to one embodiment, the electronic device can identify feature points (e.g., F11, F12, F13, F14, F15) included in the region 140-1 based on identifying the region 140-1 of the first image 120-1. The electronic device can identify feature points (e.g., F11, F12, F13, F14, F15) based on color and / or brightness changes between pixels included in the first image 120-1. For example, the feature points extracted from the region 140-1 can represent pixels with relatively abrupt changes in color and / or brightness. For example, the feature points can include edges, corners, and / or skeletons represented by pixels included in the region 140-1.

[0060] Furthermore, according to one embodiment, the electronic device can determine whether an external object present in region 140-1 of first image 120-1 is present in an image (acquired) located after first image 120-1 (after time t has elapsed) by tracking the motion trajectory of an external object contained in region 140-1 via a Pixel Trajectory Estimation method that estimates the trajectory of pixels (Pixels) present in image frames constituting video 120, and predict the position of the present object.

[0061] According to an embodiment, the electronic device can extract feature points from an Nth image 120-N at a second time point after a first time point of a first image 120-1 in a sequence of images of a video 120. The electronic device can compare feature points F11, F12, F13, F14, F15 extracted from a region 140-1 of the first image 120-1 with feature points extracted from the Nth image 120-N to identify a region 140-N in the Nth image 120-N that has a similar color and / or brightness to the region 140-1. For example, the electronic device can identify feature points F21, F22, F23, F24, F25, F26 from the Nth image 120-N that are similar to feature points F11, F12, F13, F14, F15 in the region 140-1. Based on identifying feature points F21, F22, F23, F24, F25, and F26 from the Nth image 120-N, the electronic device can determine region 140-N containing feature points F21, F22, F23, F24, F25, and F26 and / or visual object 130-N contained in region 140-N as visual objects related to the external object identified through region 140-1 of the first image 120-1.

[0062] Although one embodiment has been described in which the electronic device compares feature points F11, F12, F13, F14, F15 included in the region 140-1 of the first image 120-1 including a first location corresponding to an external object with one or more feature points included in the Nth image 120-N to identify a second location associated with an external object in the Nth image 120-N, the embodiment is not limited thereto. According to one embodiment, the electronic device can input the first image 120-1 and the Nth image 120-N into a model for recognizing the external object to identify the first location and the second location.

[0063] As described above, according to one embodiment, the electronic device can extract images (e.g., N-th image 120-N) corresponding to one or more timings spaced apart along a specified time interval from a first timing of the first image 120-1 corresponding to the region 140-1 identified by a user input (e.g., the input described with reference to FIG. 1b) in a plurality of images of the video 120. The electronic device can compare the feature points of the extracted images with the feature points F11, F12, F13, F14, F15 in the region 140-1 of the first image 120-1 to identify a position associated with an external object (e.g., an external object associated with the region 140-1) in each of the extracted images. The positions identified by the electronic device can be stored as labeling information corresponding to the video 120. According to one embodiment, the electronic device can repeatedly perform a first type of calculation for recognizing an external object based on the feature points for each of the time intervals from the first timing of the first image 120-1. By repeatedly performing the first type of calculation, the electronic device can reduce the user input (e.g., the input described with reference to FIG. 1b) required to obtain the labeling information. Based on linear interpolation for the positions identified by the first type of calculation, the electronic device can obtain the labeling information of the entire plurality of images included in the video 120. In the following, an example of the operation of the electronic device that obtains the labeling information based on linear interpolation will be described with reference to FIGS. 5a to 5b.

[0064] 5a-5b show an example of an operation of an electronic device that obtains labeling information of another image based on labeling information of two images included in a video. The operation of the electronic device described with reference to Figs. 5a-5b may be performed by the electronic device 101 and / or the processor 210 of Figs. 1a-1b and 2. The operation of the electronic device described with reference to Figs. 5a-5b may be related to at least one of the operations (e.g., operation 340) of Fig. 3.

[0065] Referring to Figure 5a, there is shown regions 140-1, 140-N corresponding to a first image 120-1 and an Nth image 120-N, respectively, identified by an electronic device having performed the operations described with reference to Figure 4. According to one embodiment, one or more feature points in the first image 120-1 may be compared with one or more feature points in the Nth image 120-N to identify regions 140-1, 140-N corresponding to external objects associated with all of the first image 120-1 and the Nth image 120-N.

[0066] 5a, for convenience of explanation, the positions of the regions 140-1, 140-N identified from the first image 120-1 and the Nth image 120-N are superimposed and displayed in a coordinate plane 510. The vertices of the coordinate plane 510 may be matched to the vertices of the first image 120-1 and the Nth image 120-N, respectively. According to an embodiment, the electronic device may obtain labeling information of an image (e.g., the kth image 120-k) between the first image 120-1 and the Nth image 120-N based on the positions of the regions 140-1, 140-N. For example, the labeling information obtained by the electronic device may include a position corresponding to the kth image 120-k between the first image 120-1 and the Nth image 120-N and related to an external object.

[0067] 5a, a region 520 in the kth image 120-k is shown, which is obtained based on the interpolation of the positions of the regions 140-1, 140-N. According to one embodiment, the electronic device can obtain the coordinates of the vertices AK, BK, CK, and DK of the region 520 based on Equation 1.

[0068]

number

[0069] Referring to the formula 1, the x-coordinate (x AK ) are the x-coordinates (x A1 , x AN Similarly, the y coordinate (y AK ) are the y coordinates (y A1 , y AN ) may have a value that divides the vertices A, B, C, D of the region 520 into N:k. For example, each of the vertices A, B, C, D of the region 520 may correspond to an internal division point that divides each of the vertices A, B, C, D of the region 140-1 and the vertices A, B, N, C, D of the region 140-N into N:k. Referring to FIG. 5a, each of the vertices A, B, C, D of the region 520 may be located on a line that connects each of the vertices A, B, N, C, D of the region 140-1 to each of the vertices A, B, N, C, D of the region 140-N. N in the number 1 may indicate the length of the time interval between the first image 120-1 and the Nth image 120-N, and k in the number 1 may indicate the position within the time interval of the timing corresponding to the kth image 120-k.

[0070] 5b, the operation of the electronic device 101 to identify a region 520 from the k-th image 120-k is exemplarily described. The electronic device 101 can obtain the coordinates of vertices AK, BK, CK, and DK of the region 520 of the k-th image 120-k between the first image 120-1 and the N-th image 120-N based on the formula 1. The x-coordinate of the vertex AK of the region 520 in FIG. 5b may have a value that divides the x-coordinate of the vertex A1 of the region identified from the first image 120-1 and the x-coordinate of the vertex AN of the N-th image 120-N into N:k. Similarly, the y-coordinate of the vertex AK of the region 520 may have a value that divides the y-coordinate of the vertex A1 of the region identified from the first image 120-1 and the y-coordinate of the vertex AN of the N-th image 120-N into N:k. The x-coordinates of the other vertices BK, CK, DK of region 520 may also have values ​​that divide the x-coordinates of the vertices B1, C1, D1 of the region identified from the first image 120-1 and the x-coordinates of the vertices BN, CN, DN of the region identified from the Nth image 120-N into N:k. Similarly, the y-coordinates of the other vertices BK, CK, DK of region 520 may also have values ​​that divide the y-coordinates of the vertices B1, C1, D1 of the region identified from the first image 120-1 and the y-coordinates of the vertices BN, CN, DN of the region identified from the Nth image 120-N into N:k.

[0071] According to an embodiment, the electronic device may store data indicative of the positions and / or sizes of the regions 140-1, 520, 140-N as labeling information corresponding to the video 120. For example, the electronic device may store data on vertices A1, B1, C1, D1 as information indicative of the region 140-1 corresponding to the first image 120-1 in the labeling information. The electronic device may store data indicative of the coordinates of vertices AK, BK, CK, DK as information indicative of the region 140-k corresponding to the k-th image 120-k in the labeling information. The electronic device may store parameters associated with at least one of the vertices AN, BN, CN, DN as information indicative of the region 140-N corresponding to the N-th image 120-N in the labeling information. The electronic device may further store parameters indicative of the correspondence of these parameters to the N-th image 120-N together with parameters associated with at least one of the vertices AN, BN, CN, DN of the region 140-N in the labeling information.

[0072] As described above, according to one embodiment, the electronic device can obtain a third coordinate indicating a third position of the region 520 in the k-th image 120-k between the first image 120-1 and the N-th image 120-N by interpolating a first coordinate indicating a first position of the region 140-1 in the first image 120-1 at a first timing and a second coordinate indicating a second position of the region 140-N in the N-th image 120-N at a second timing using a length between the first timing and the second timing (e.g., N in the number 1). The electronic device can obtain the third position by interpolating the first coordinate and the second coordinate based on the timing of the k-th image 120-k (e.g., k in the number 1). For example, the electronic device can perform a second type of calculation (e.g., the interpolation described with reference to the number 1) to obtain the third position based on the first position, the second position, and the timing of the k-th image.

[0073] In the following, an exemplary operation of the electronic device described with reference to the previous figures will be described with reference to FIG.

[0074] 6 illustrates an example of a flow chart of operations performed by an electronic device, according to one embodiment. The operations of the electronic device described with reference to FIG. 6 may be performed by the electronic device 101 and / or the processor 210 of FIGS. 1a-1b and 2. The operations of the electronic device described with reference to FIG. 6 may be related to the operations of the electronic device described with reference to FIG. 3.

[0075] Referring to Figure 6, in operation 610, according to one embodiment, a processor of an electronic device may identify a plurality of images for playing a video. The processor may perform operation 610 of Figure 6 similarly to operation 310 of Figure 3. The video of operation 610 may include video 120 of Figures 1a-5a.

[0076] Referring to FIG. 6, in an operation 620, a processor of an electronic device, according to an embodiment, can obtain labeling data corresponding to an a-th image of a plurality of images. The a-th image of operation 620 can include the first image 120-1 of FIGS. 1a-5b. The labeling data of operation 620 can represent a region in the a-th image associated with an external object (e.g., region 140-1 of FIGS. 1a-5a). The processor can obtain the labeling data based on identifying an input indicating a selection of the region in the a-th image via the screen 110 of FIG. 1a. The input can include the input described with reference to FIG. 1b. The processor can obtain the labeling data based on an input to a location of the region (e.g., coordinates of vertices A1, B1, C1, D1 of region 140-1 of FIGS. 1a-5a).

[0077] Referring to FIG. 6, in operation 630, according to one embodiment, a processor of the electronic device may obtain labeling data corresponding to the a+Mth image of the plurality of images. The a+Mth image of operation 630 may include the Nth image 120-N of FIGS. 1a-5b. The labeling data of operation 630 obtained by the processor may represent a region in the a+Mth image associated with an external object (e.g., region 140-N of FIGS. 1a-5a). According to one embodiment, the processor may perform operation 630 of FIG. 6 based on the operations described with reference to FIG. 4.

[0078] For example, within operation 632 of Figure 6, according to one embodiment, a processor of the electronic device may compare feature points of the ath image with feature points of the a+Mth image to identify a location in the a+Mth image of an external object associated with the ath image. Based on the similarity of the ath image and the a+Mth image represented by at least one feature point, the processor may identify a respective location in the ath image and the a+Mth image of an external object included in both the ath image and the a+Mth image. In one embodiment, if a location in the a+Mth image associated with an external object cannot be automatically identified based on the feature points, the processor may request a user of the electronic device to input said location.

[0079] For example, in operation 634 of FIG. 6, a processor of the electronic device, according to an embodiment, can identify whether a visual object corresponding to an external object is included in the a+Mth image. For example, by comparing one or more feature points identified from a position in the ath image associated with the external object represented by the labeling data of operation 620 with one or more feature points included in the a+Mth image, the processor can identify whether a visual object corresponding to an external object is included in the a+Mth image. In an embodiment using a model for recognizing an external object, the processor can identify whether a visual object corresponding to an external object is included in the a+Mth image based on a probability that the a+Mth image includes an external object identified by the model. If the visual object is included in the a+Mth image (634-YES), the processor can perform operation 640. If the visual object is not included in the a+Mth image (634-NO), the processor can perform operation 636.

[0080] For example, in operation 636 of Figure 6, according to one embodiment, a processor of the electronic device can obtain labeling data corresponding to the a+Mth image based on an input indicating a selection of a position in the a+Mth image associated with an external object. In operation 636, the processor can display on the display a screen for obtaining a position associated with an external object in the a+Mth image. Based on the input received through the screen, the processor can obtain the labeling data of operation 636.

[0081] Referring to FIG. 6, within operation 640, according to one embodiment, a processor of an electronic device can generate labeling data corresponding to the (a + N)-th image based on the labeling data corresponding to the a-th image and the labeling data corresponding to the (a + M)-th image. N can be a natural number greater than 0 and less than M (0 < N < M). The (a + N)-th image of operation 640 can include the k-th image of FIGS. 1a to 5b. The processor can perform interpolation on the position within the a-th image represented by the labeling data of operation 620 and the position within the (a + M)-th image indicated by the labeling data of operation 630 to generate the labeling data corresponding to the (a + N)-th image. The processor can execute operation 640 of FIG. 6 based on the operations described with reference to FIGS. 5a and / or 5b. The processor can perform the interpolation described with reference to FIG. 1 to generate the labeling data corresponding to the (a + N)-th image.

[0082] According to one embodiment, a processor of an electronic device can execute operations 610 to 640 of FIG. 6 to obtain the labeling data of the entire a-th image to the (a + M)-th image. Referring to FIG. 6, within operation 650, according to one embodiment, a processor of an electronic device can perform calibration on the labeling data generated by operations 610 to 640. The calibration performed by the processor can be executed to combine the non-linear movement of an external object captured by video with the labeling data linearly adjusted by linear interpolation.

[0083] For example, within operation 652 of FIG. 6, according to one embodiment, a processor of an electronic device can identify whether a parameter (e.g., a natural number M) representing an interval between the ath image and the a+Mth image exceeds a threshold value. The interval can be related to the length of the time interval between the timing of the ath image and the timing of the a+Mth image. If the interval exceeds the threshold value (652-YES), the processor can perform operation 654. For example, if the interval is longer than a threshold interval, the processor can perform operation 654. If the interval is equal to or less than the threshold value (652-NO), the processor can perform operation 656.

[0084] For example, in operation 654 of FIG. 6, according to one embodiment, a processor of the electronic device may modify labeling data corresponding to the a+Nth image based on the feature points. For example, the processor may modify the labeling data corresponding to the a+Nth image based on the labeling data corresponding to the ath image (e.g., labeling data obtained from a user of the electronic device) by comparing one or more feature points extracted from the ath image with feature points included in the a+Nth image. The one or more feature points extracted from the ath image may be extracted from a region associated with an external object in the ath image represented by the labeling data. The feature points in the a+Nth image compared to the one or more feature points extracted from the ath image may be extracted from the entire a+Nth image.

[0085] For example, the processor may identify regions in the a+Nth image that are similar to regions of the ath image represented by the labeling data corresponding to the ath image based on the comparison. The processor may modify the labeling data corresponding to the a+Nth image based on locations of regions in the a+Nth image identified based on the comparison. The processor may modify the labeling data generated by operation 640 relating to interpolation based on operation 654 relating to comparing feature points of different images.

[0086] For example, in operation 656 of FIG. 6, according to one embodiment, a processor of the electronic device can modify the labeling data based on an input received via a screen for displaying the video and the labeling data. The screen of operation 656 can include screen 110 of FIG. 1a. The processor of the electronic device can identify, via the screen, an input for modifying the labeling data corresponding to any of the ath image through the a+Mth image. Based on this input, the processor can adjust the labeling data corresponding to the input to modify a position in the image relative to an external object.

[0087] 6, in operation 660, according to one embodiment, a processor of the electronic device may store labeling data for the ath image, the a+Nth image, and the a+Mth image in association with the video. The labeling data stored in association with the video by operation 660 may correspond to the labeling information 240 of FIG. 2. The labeling data stored by operation 660, together with the video in association with the labeling data, may be used to train a model (or neural network) for recognizing the video and / or images.

[0088] In one embodiment, the labeling data identified based on operation 660 may include information such as that in Table 1.

[0089] [Table 1]

[0090] The information on the external object in Table 1 (Shapes) may include information on the external object included in the image as a sub-object. The information on the external object in Table 1 may include information as in Table 2.

[0091] [Table 2]

[0092] In one embodiment, K-fold cross validation can be performed for training the neural network. Based on FIG. 6, the processor that has acquired labeling data corresponding to a large amount of images can execute an algorithm for verifying the reliability of the neural network, such as K-fold cross validation. For example, the processor can divide the labeling data into a training set, a validation set, and / or a test set. For example, the processor can distinguish the labeling data based on the image flags in Table 1. For example, based on the image flags based on the weather, the processor can classify the labeling data according to the weather. Classifying the labeling data based on the image flags can be used to uniformize the characteristics of the images used for training and / or validation of the neural network. For example, using the labeling data classified based on the image flags according to the weather, the processor can sample the labeling data from a group of labeling data corresponding to different weathers, such that the labeling data classified into a specific weather is not used for training and / or validation of the neural network. If the labeling data is generated for more than the specified number of images, the processor may divide the labeling data into three sets: a training set, a validation set, and a test set. If the labeling data is generated for less than the specified number of images, the processor may divide the labeling data into two sets: a training set and a test set.

[0093] In one embodiment, the processor can divide the labeling data included in the training set into K folds. In the divided folds, the processor can designate K-1 labeling data as the labeling data for training and the remaining one labeling data as the labeling data for validation after dividing the labeling data into K folds again. The processor can generate a neural network (or model), input the labeling data for training, and extract an error value. The processor can extract an error value while crossing the labeling data for validation for each fold. After extracting the error values ​​of all folds, the processor can perform optimization of the neural network based on the extracted error value. The processor can perform training on the entire training set based on the optimized neural network. After performing training on the entire training set, the processor can perform evaluation of the neural network based on the labeling data included in the test set.

[0094] As described above, according to one embodiment, the processor of the electronic device can alternately perform the first type of calculation based on the comparison of feature points and the second type of calculation based on the interpolation between the positions (or coordinates) indicated by the labeling data to obtain the labeling data for all of the multiple images included in the video. Since the processor alternates the calculations, the processor can obtain the labeling data more quickly without extracting feature points from all of the multiple images.

[0095] An example of the operation of an electronic device having identified a user input for modifying labeling data will now be described with reference to FIG.

[0096] Figure 7 illustrates an example of an interpolation performed by an electronic device, according to one embodiment. The operations of the electronic device described with reference to Figure 7 may be performed by the electronic device 101 and / or the processor 210 of Figures 1a-1b and 2. The operations of the electronic device described with reference to Figure 7 may relate to at least one of the operations of Figures 3 and / or 6 (e.g., at least one of operations 654, 656 of Figure 6).

[0097] Referring to FIG. 7, regions 140-1, 520, 140-N corresponding to a first image 120-1, a k-th image 120-k, and an N-th image 120-N included in a sequence of a plurality of images for playing a video 120 are shown. The region 140-1 corresponding to the first image 120-1 can be identified by a user input (e.g., the input described with reference to FIG. 1b) for selecting a position related to an external object in the first image 120-1. The region 140-N corresponding to the N-th image 120-N can be identified based on a comparison between one or more feature points included in the region 140-1 of the first image 120-1 and feature points included in the N-th image 120-N. The region 520 corresponding to the k-th image 120-k can be obtained based on the interpolation described with reference to FIGS. 5a-5b. For example, the electronic device can identify the position of the region 520 of the k-th image 120-k based on the interpolation of the regions 140-1, 140-N.

[0098] In the exemplary case of FIG. 7, the position of the region 520 linearly obtained by interpolation may be separated from the position associated with the external object in the k-th image 120-k based on the non-linear movement of the external object represented by the video 120. According to an embodiment, the processor of the electronic device may change the region 520 identified by the interpolation based on the operation 654 of FIG. 6 by comparing the feature points included in the k-th image 120-k with the feature points of other images (e.g., the first image 120-1) (e.g., the feature points in the region 140-1 of the first image 120-1). With reference to FIG. 7, the processor may change the region associated with the external object in the k-th image 120-k from the region 520 to the region 720 based on the comparison of the feature points. The processor may change and / or move the region 520 obtained by interpolation to the region 720 based on the operation 654 of FIG. 6.

[0099] In the exemplary case of FIG. 7, the processor can identify an input indicating a change of the region 520 for the kth image 120-k. For example, while displaying the kth image 120-k via a display (e.g., the display 230 of FIG. 2), the processor can display a visual object (e.g., a bounding box) superimposed on the kth image 120-k indicating a position of an external object based on the labeling information corresponding to the kth image 120-k. The position of the visual object on the kth image 120-k can correspond to the region 520. In the above example, the processor can identify an input representing a movement of the visual object. Based on this input, the processor can adjust the labeling data corresponding to the kth image 120-k based on the position of the visual object moved by the input. For example, based on the input moving the region 520 to the region 720, the processor can change the position related to the external object in the kth image 120-k in the labeling data to the position of the region 720.

[0100] According to an embodiment, the processor of the electronic device may, based on changing the position related to the external object in the kth image 120-k, modify the labeling data of other images adjacent to the kth image 120-k in the sequence of the plurality of images for playing the video 120 with the modified position. For example, upon changing the position related to the external object in the kth image 120-k from the position related to the region 520 to the position related to the region 720, the processor may modify the position related to the external object in at least one image different from the kth image 120-k. For example, the processor may modify the labeling data corresponding to other images between the first image 120-1 and the kth image 120-k (e.g., the k-1st image 120-k-1) and / or other images between the kth image 120-k and the Nth image 120-N (e.g., the k+1st image 120-k+1).

[0101] 7, with the region 520 of the kth image 120-k represented by the labeling data obtained based on the interpolation changed to the region 720, the processor can identify positions and / or regions related to external objects in other images between the first image 120-1 and the kth image 120-k on the line connecting the vertices AK2, BK2, CK2, and DK2 of the region 720 and the vertices A1, B1, C1, and D1 of the first image 120-1. For example, the processor can obtain the labeling data of the images between the first image 120-1 and the kth image 120-k based on the interpolation of the region 140-1 of the first image 120-1 and the region 720 of the kth image 120-k. Similarly, in this state, the processor can identify locations and / or regions associated with external objects from other images between the kth image 120-k and the Nth image 120-N on lines connecting vertices AK2, BK2, CK2, DK2 of region 720 to vertices AN, BN, CN, DN of the Nth image 120-N.

[0102] As described above, based on the input for changing the region 520 of the kth image 120-k to the region 720, which is represented by the labeling data obtained based on the interpolation, the electronic device 101 can further adjust the labeling data of other images adjacent to the kth image 120-k (e.g., images between the first image 120-1 and the kth image 120-k and / or other images between the kth image 120-k and the Nth image 120-N). Based on this adjustment, the electronic device 101 can use the input to improve the accuracy of the labeling information for the multiple images included in the video 120.

[0103] As described above, according to one embodiment, the electronic device can selectively apply feature point matching and / or linear interpolation to multiple images in order to more quickly obtain labeling information for the multiple images included in the video 120. The electronic device can maintain and / or improve the accuracy of the labeling information based on the feature point matching while reducing the amount of calculation and / or time for generating the labeling information based on the interpolation.

[0104] FIG. 8 illustrates an example block diagram illustrating an autonomous driving system for a vehicle, according to one embodiment.

[0105] 9 and 10 show an example of a block diagram illustrating an autonomous vehicle, according to one embodiment.

[0106] FIG. 11 illustrates an example of a gateway associated with a user device, according to one embodiment.

[0107] FIG. 8 illustrates an example block diagram illustrating an autonomous driving system for a vehicle, according to one embodiment.

[0108] The vehicle autonomous driving system 800 according to FIG. 8 may be a deep learning network including a sensor 803, an image preprocessor 805, a deep learning network 807, an artificial intelligence (AI) processor 809, a vehicle control module 811, a network interface 813, and a communication unit 815. In various embodiments, each element may be connected via various interfaces. For example, sensor data sensed and output by the sensor 803 may be fed to the image preprocessor 805. The sensor data processed by the image preprocessor 805 may be fed to the deep learning network 807 run by the AI ​​processor 809. The output of the deep learning network 807 run by the AI ​​processor 809 may be fed to the vehicle control module 811. An intermediate result of the deep learning network 807 run by the AI ​​processor 809 may be fed to the AI ​​processor 809. In various embodiments, the network interface 813 communicates with an in-vehicle electronic device to transmit autonomous driving path information and / or autonomous driving control commands for autonomous driving of the vehicle to the internal block configuration. In one embodiment, the network interface 813 may be used to transmit sensor data acquired via the sensor 803 to an external server. In some embodiments, the autonomous driving control system 800 may include additional or fewer components as appropriate. For example, in some embodiments, the image preprocessor 805 may be an optional component. In another example, a post-processing component (not shown) may be included in the autonomous driving control system 800 to perform post-processing on the output of the deep learning network 807 before the output is provided to the vehicle control module 811.

[0109] In some embodiments, the sensor 803 may include one or more sensors. In various embodiments, the sensor 803 may be mounted at different locations on the vehicle. The sensor 803 may be oriented in one or more different directions. For example, the sensor 803 may be mounted on the front, sides, rear, and / or roof of the vehicle to be oriented in a forward-facing, rear-facing, side-facing, etc. direction. In some embodiments, the sensor 803 may be an image sensor, such as high dynamic range cameras. In some embodiments, the sensor 803 includes non-visual sensors. In some embodiments, the sensor 803 includes RADAR, LiDAR (Light Detection And Ranging), and / or ultrasonic sensors in addition to the image sensor. In some embodiments, the sensor 803 is not mounted on the vehicle with the vehicle control module 811. For example, sensors 803 may be included as part of the deep learning system to capture sensor data and may be mounted in the environment or road and / or mounted in surrounding vehicles.

[0110] In some embodiments, the image pre-processor 805 can be used to pre-process the sensor data of the sensor 803. For example, the image pre-processor 805 can be used to pre-process the sensor data, to split the sensor data into one or more components, and / or to post-process the one or more components. In some embodiments, the image pre-processor 805 can be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image pre-processor 805 can be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image pre-processor 805 can be a component of the AI ​​processor 809.

[0111] In some embodiments, the deep learning network 807 may be a deep learning network for implementing control instructions for controlling an autonomous vehicle. For example, the deep learning network 807 may be an artificial neural network, such as a convolutional neural network (CNN), trained using sensor data, and the output of the deep learning network 807 is provided to the vehicle control module 811.

[0112] In some embodiments, the artificial intelligence (AI) processor 809 may be a hardware processor for running the deep learning network 807. In some embodiments, the AI ​​processor 809 is a specialized AI processor for performing inference via a convolutional neural network (CNN) on the sensor data. In some embodiments, the AI ​​processor 809 may be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 809 may be optimized for deep learning operations such as neural network operations including convolution, dot product, vector, and / or matrix operations. In some embodiments, the AI ​​processor 809 may be implemented via multiple graphics processing units (GPUs) capable of efficiently performing parallel processing.

[0113] In various embodiments, the AI ​​processor 809 can be coupled via an input / output interface to a memory configured to provide the AI ​​processor with instructions that, while the AI ​​processor 809 is running, cause the AI ​​processor 809 to perform deep learning analysis on the sensor data received from the sensors 803 and determine the results of the machine learning used to operate the vehicle at least partially autonomously. In some embodiments, a Vehicle Control Module 811 can be used to process instructions for vehicle control output from the Artificial Intelligence (AI) processor 809 and translate the output of the AI ​​processor 809 into instructions for controlling the various modules of the vehicle to control each of the modules of the vehicle. In some embodiments, the Vehicle Control Module 811 is used to control the vehicle for autonomous driving. In some embodiments, the Vehicle Control Module 811 can adjust the steering and / or speed of the vehicle. For example, the Vehicle Control Module 811 can be used to control the driving of the vehicle, such as deceleration, acceleration, steering, lane changing, lane keeping, etc. In some embodiments, the vehicle control module 811 can generate control signals to control vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 811 can be used to control vehicle audio-related systems, such as the vehicle's sound system, the vehicle's audio warnings, the vehicle's microphone system, the vehicle's horn system, etc.

[0114] In some embodiments, the vehicle control module 811 can be used to control notification systems, including warning systems to inform passengers and / or drivers of access to an intended destination or of a driving event, such as a potential collision. In some embodiments, the vehicle control module 811 can be used to adjust sensors, such as the sensor 803, of the vehicle. For example, the vehicle control module 811 can modify the orientation of the sensor 803, change the output resolution and / or format type of the sensor 803, increase or decrease the capture rate, adjust the dynamic range, adjust the focus of the camera. Additionally, the vehicle control module 811 can turn on / off the operation of the sensors, individually or collectively.

[0115] In some embodiments, the vehicle control module 811 can be used to modify parameters of the image preprocessor 805, such as by changing the range of filter frequencies, adjusting edge detection parameters for feature and / or object detection, adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 811 may be used to control the autonomous driving of the vehicle and / or Driver assistance functions of the vehicle.

[0116] In some embodiments, the network interface 813 may be responsible for an internal interface between the block configuration of the autonomous driving control system 800 and the communication unit 815. Specifically, the network interface 813 may be a communication interface for receiving and / or transmitting data including voice data. In various embodiments, the network interface 813 may connect to an external server via the communication unit 815 to connect a voice call, receive and / or transmit a text message, transmit sensor data, update software of the vehicle in the autonomous driving system, or update software of the autonomous driving system of the vehicle.

[0117] In various embodiments, the communication unit 815 can include various wireless interfaces, such as cellular or WiFi. For example, the network interface 813 can be used to receive updates to operational parameters and / or instructions for the sensor 803, the image preprocessor 805, the deep learning network 807, the AI ​​processor 809, and the vehicle control module 811 from an external server connected via the communication unit 815. For example, the machine learning model of the deep learning network 807 can be updated using the communication unit 815. According to yet another example, the communication unit 815 can be used to update operational parameters of the image preprocessor 805, such as image processing parameters, and / or firmware of the sensor 803.

[0118] In another embodiment, the communication unit 815 can be used to activate communications for emergency services and emergency contact in the event of an accident or near-accident. For example, in a crash event, the communication unit 815 can be used to call emergency services for assistance and to notify the emergency services of the crash details and vehicle location to the outside world. In various embodiments, the communication unit 815 can update or obtain an expected time of arrival and / or a destination location.

[0119] According to an embodiment, the autonomous driving system 800 shown in Fig. 8 may be configured in an electronic device of a vehicle. According to an embodiment, when an autonomous driving cancellation event occurs from a user during autonomous driving of the vehicle, the AI ​​processor 809 of the autonomous driving system 800 can control the autonomous driving software of the vehicle to learn by controlling the input of autonomous driving cancellation event related information into the training set data of the deep learning network.

[0120] 9 and 10 show an example of a block diagram illustrating an autonomous vehicle according to an embodiment. Referring to FIG. 9, an autonomous vehicle 900 according to the present embodiment may include a control device 1000, sensing modules 904a, 904b, 904c, and 904d, an engine 906, and a user interface 908.

[0121] The autonomous vehicle 900 may have an autonomous mode or a manual mode. As an example, the autonomous vehicle 900 may switch from the manual mode to the autonomous mode or from the autonomous mode to the manual mode according to user input received via the user interface 908.

[0122] When the vehicle 900 is operated in an autonomous driving mode, the autonomous driving vehicle 900 can operate under the control of the control device 1000.

[0123] In this embodiment, the control device 1000 may include a controller 1020 including a memory 1022 and a processor 1024 , a sensor 1010 , a communication device 1030 , and an object detection device 1040 .

[0124] Here, the object detection device 1040 can perform all or part of the functions of a distance measurement device (eg, the electronic device 101).

[0125] That is, in this embodiment, the object detection device 1040 is a device for detecting objects located outside the moving body 900, and the object detection device 1040 can detect objects located outside the moving body 900 and generate object information according to the detection results.

[0126] The object information can include information regarding the presence or absence of an object, position information of the object, distance information between the moving body and the object, and relative speed information between the moving body and the object.

[0127] The object may include various objects located outside the moving body 900, such as a lane, another vehicle, a pedestrian, a traffic signal, a light, a road, a structure, an overspeed prevention jaw, a topographical object, and an animal. Here, the traffic signal may be a concept including a traffic light, a traffic sign, and a pattern or text painted on the road surface. And, the light may be light generated from a lamp provided on another vehicle, light generated by a street lamp, or sunlight.

[0128] And the structures may be objects located around the road and fixed to the ground. For example, the structures may include street lights, roadside trees, buildings, utility poles, traffic lights, bridges. The terrain objects may include mountains, hills, etc.

[0129] Such an object detection device 1040 may include a camera module, and the controller 1020 may extract object information from an external image captured by the camera module and process the information.

[0130] The object detection device 1040 may further include an imaging device for recognizing the external environment. In addition to LIDAR, RADAR, GPS devices, odometry and other computer vision devices, ultrasonic sensors, infrared sensors, etc. may be used, which may be selected or operated simultaneously as needed to enable more accurate sensing.

[0131] On the other hand, a distance measuring device according to one embodiment of the present invention can calculate the distance between an autonomous vehicle 900 and an object, and control the operation of the vehicle based on the calculated distance in cooperation with a control device 1000 of the autonomous vehicle 900.

[0132] As an example, when there is a possibility of a rear-end collision depending on the distance between the autonomous traveling vehicle 900 and the object, the autonomous traveling vehicle 900 can control the brakes to reduce or stop the vehicle. As another example, when the object is a moving object, the autonomous traveling vehicle 900 can control the traveling speed of the autonomous traveling vehicle 900 to maintain a predetermined distance or more from the object.

[0133] The distance measurement device according to one embodiment of the present invention can be configured as one module in the control device 1000 of the autonomous vehicle 900. That is, the memory 1022 and the processor 1024 of the control device 1000 can realize the collision prevention method according to the present invention in software.

[0134] Also, the sensor 1010 can acquire various sensing information by connecting the internal / external environment of the moving object to the sensing modules 904a, 904b, 904c, and 904d. Here, the sensor 1010 can include an attitude sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor, a collision sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight detection sensor, a heading sensor, a gyro sensor, a position module, a moving object forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on the rotation of a steering wheel, a moving object internal temperature sensor, a moving object internal humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0135] As a result, the sensor 1010 can acquire sensing signals for moving body posture information, moving body collision information, moving body direction information, moving body position information (GPS information), moving body angle information, moving body speed information, moving body acceleration information, moving body inclination information, moving body forward / reverse information, battery information, fuel information, tire information, moving body lamp information, moving body internal temperature information, moving body internal humidity information, steering wheel rotation angle, moving body external illuminance, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.

[0136] In addition, the sensor 1010 may further include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), and the like.

[0137] In this manner, the sensor 1010 can generate mobile object status information based on the sensing data.

[0138] The wireless communication device 1030 is configured to perform wireless communication between the autonomously traveling mobile body 900. For example, the wireless communication device 1030 enables the autonomously traveling mobile body 900 to communicate with a user's mobile phone, another wireless communication device 1030, another mobile body, a central device (traffic control device), a server, etc. The wireless communication device 1030 can transmit and receive wireless signals according to a connection wireless protocol. The wireless communication protocols are Wi-Fi, Bluetooth, LTE (Long-Term Evolution), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and GSM (Global Systems for Mobile Communications), and the communication protocol is not limited thereto.

[0139] In addition, the autonomous traveling mobile body 900 in this embodiment can also realize communication between mobile bodies via the wireless communication device 1030. That is, the wireless communication device 1030 can communicate with other mobile bodies on the road and other mobile bodies by vehicle-to-vehicle (V2V) communication. The autonomous traveling mobile body 900 can transmit and receive information such as driving warnings and traffic information by vehicle-to-vehicle communication, and can also request information from other mobile bodies or receive requests. For example, the wireless communication device 1030 can perform V2V communication by a short-range communication (DSRC, dedicated short-range communication) device or a C-V2V (Cellular-V2V) device. In addition to communication between vehicles, communication between vehicles and other objects (e.g., electronic devices carried by pedestrians, etc.) (V2X, Vehicle to Everything communication) can also be realized via the wireless communication device 1030.

[0140] In this embodiment, the controller 1020 is a unit that controls the overall operation of each unit in the moving body 900, and may be configured by the manufacturer of the moving body at the time of manufacture, or may be further configured to execute an autonomous driving function after manufacture. Alternatively, the controller 1020 may include a configuration for executing a continuous additional function through an upgrade of the controller 1020 configured at the time of manufacture. Such a controller 1020 may be called an ECU (Electronic Control Unit).

[0141] The controller 1020 can collect various data from the connected sensors 1010, the object detection device 1040, the communication device 1030, etc., and transmit control signals based on the collected data to the sensors 1010, the engine 906, the user interface 908, the communication device 1030, the object detection device 1040, and other components in the vehicle. In addition, although not shown, the controller 1020 can also transmit control signals to an acceleration device, a braking system, a steering device, or a navigation device related to the traveling of the vehicle.

[0142] In this embodiment, the controller 1020 can control the engine 906, for example, by detecting the speed limit of the road on which the autonomous vehicle 900 is traveling, and controlling the engine 906 so that the traveling speed does not exceed the speed limit, or by controlling the engine 906 to accelerate the traveling speed of the autonomous vehicle 900 within a range that does not exceed the speed limit.

[0143] In addition, when the autonomous traveling vehicle 900 approaches a lane or deviates from a lane during the traveling of the autonomous traveling vehicle 900, the controller 1020 can determine whether such approach and departure from the lane correspond to a normal traveling situation or other traveling situation, and control the engine 906 to control the traveling of the moving body according to the determination result. Specifically, the autonomous traveling vehicle 900 can detect lanes formed on both sides of the road on which the moving body is traveling. In this case, the controller 1020 can determine whether the autonomous traveling vehicle 900 approaches a lane or deviates from the lane, and when it is determined that the autonomous traveling vehicle 900 approaches a lane or deviates from the lane, it can determine whether such traveling is due to a correct traveling situation or due to other traveling situations. Here, an example of a normal traveling situation may be a situation in which a lane change of the moving body is required. And, an example of another traveling situation may be a situation in which a lane change of the moving body is not required. When the controller 1020 determines that the autonomously driving vehicle 900 is close to a lane or has left the lane in a situation where the vehicle does not need to change lanes, the controller 1020 can control the driving of the autonomously driving vehicle 900 so that the autonomously driving vehicle 900 does not leave the lane and drives normally.

[0144] If there is another moving body or an obstacle ahead of the moving body, the engine 906 or the braking system can be controlled to decelerate the moving body, and in addition to the speed, the trajectory, the travel path, and the steering angle can be controlled. Alternatively, the controller 1020 may generate necessary control signals to control the traveling of the moving body according to recognition information of other external environments such as the traveling lane of the moving body, the traveling signal, etc.

[0145] In addition to generating its own control signals, the controller 1020 can also communicate with surrounding mobile objects or a central server and, via the information received, control the movement of the mobile object by sending commands to control the peripheral devices.

[0146] In addition, when the position of the camera module 1050 is changed or the angle of view is changed, accurate recognition of the moving body or lane according to the present embodiment may be difficult. To prevent this, the controller 1020 may generate a control signal to control the camera module 1050 to perform calibration. Therefore, in the present embodiment, the controller 1020 generates a calibration control signal in the camera module 1050, so that even if the mounting position of the camera module 1050 is changed due to vibration, impact, etc. generated with the movement of the autonomous traveling moving body 900, the normal mounting position, direction, angle of view, etc. of the camera module 1050 can be continuously maintained. The controller 1020 may generate a control signal to perform calibration of the camera module 1050 when the initial mounting position, direction, angle of view information of the camera module 1050 stored in advance and the initial mounting position, direction, angle of view information of the camera module 1050 measured during the movement of the autonomous traveling moving body 900 change by more than a critical value.

[0147] In this embodiment, the controller 1020 may include a memory 1022 and a processor 1024. The processor 1024 may execute software stored in the memory 1022 according to control signals of the controller 1020. In particular, the controller 1020 may store data and instructions in the memory 1022 for performing a lane detection method in accordance with the present invention, which instructions may be executed by the processor 1024 to implement one or more methods disclosed herein.

[0148] In this case, the memory 1022 may be stored in a non-volatile recording medium executable by the processor 1024. The memory 1022 can store software and data via an appropriate internal or external device. The memory 1022 can be composed of a random access memory (RAM), a read only memory (ROM), a hard disk, and a memory 1022 device connected to a dongle.

[0149] The memory 1022 may store at least an operating system (OS), user applications, and executable instructions. The memory 1022 may also store application data and array data structures.

[0150] The processor 1024 may be a microprocessor or any suitable electronic processor, controller, microcontroller, or state machine.

[0151] The processor 1024 may be implemented as a combination of computing devices, which may comprise a digital signal processor, a microprocessor, or any suitable combination thereof.

[0152] Meanwhile, the autonomous vehicle 900 may further include a user interface 908 for user input to the control device 1000 described above. The user interface 908 allows the user to input information through appropriate interaction. For example, the user interface 908 may be implemented as a touch screen, a keypad, an operation button, or the like. The user interface 908 transmits an input or command to the controller 1020, and the controller 1020 may execute a control operation of the vehicle in response to the input or command.

[0153] Furthermore, the user interface 908 is a device external to the autonomous vehicle 900 and can communicate with the autonomous vehicle 900 via a wireless communication device 1030. For example, the user interface 908 can be linked to a mobile phone, a tablet, or another computer device.

[0154] Further, in this embodiment, the autonomous vehicle 900 has been described as including the engine 906, but may include other types of propulsion systems. For example, the vehicle may be powered by electric energy, hydrogen energy, or a hybrid system that combines these. Thus, the controller 1020 may include the propulsion mechanisms of the propulsion system of the autonomous vehicle 900 and provide control signals to the configuration of each propulsion mechanism.

[0155] Hereinafter, with reference to FIG. 10, the detailed configuration of the control device 1000 according to the present embodiment will be described in more detail.

[0156] The control device 1000 includes a processor 1024. The processor 1024 may be a general purpose single-chip or multi-chip microprocessor, a special purpose microprocessor, a microcontroller, a programmable gate array, etc. The processor may be referred to as a central processing unit (CPU). Also, in this embodiment, the processor 1024 may be used in combination with multiple processors.

[0157] The controller 1000 also includes a memory 1022. The memory 1022 may be any electronic component capable of storing electronic information. The memory 1022 may also include a single memory as well as a combination of memories 1022.

[0158] Data and instructions 1022a for executing the distance measurement method of the distance measurement device according to the present invention may be stored in the memory 1022. When the processor 1024 executes the instructions 1022a, all or part of the instructions 1022a and data 1022b required for executing the instructions may be loaded 1024a, 1024b onto the processor 1024.

[0159] The control device 1000 may include a transmitter 1030a, a receiver 1030b, or a transceiver 1030c to enable transmission and reception of signals. One or more antennas 1032a, 1032b may be electrically connected to the transmitter 1030a, the receiver 1030b, or each transceiver 1030c and may further include an antenna.

[0160] The control device 1000 may include a digital signal processor (DSP) 1070. The DSP 1070 may enable the mobile unit to rapidly process digital signals.

[0161] The control device 1000 may include a communication interface 1080. The communication interface 1080 may include one or more ports and / or communication modules for connecting other devices to the control device 1000. The communication interface 1080 may allow a user to interact with the control device 1000.

[0162] The various components of the controller 1000 may be connected together by one or more buses 1090, which may also include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 1024, the components may communicate information to each other via the bus 1090 to perform desired functions.

[0163] Meanwhile, in various embodiments, the control device 1000 may be associated with a gateway for communication with the security cloud. For example, referring to FIG. 11, the control device 1000 may be associated with a gateway 1105 for providing information acquired from at least one of the components 1101 to 1104 of the vehicle 1100 to the security cloud 1106. For example, the gateway 1105 may be included in the control device 1000. In another example, the gateway 1105 may be composed of a separate device in the vehicle 1100 that is distinct from the control device 1000. The gateway 1105 communicatively connects the software management cloud 1109 having different networks, the security cloud 1106, and a network in the vehicle 1100 protected by the in-vehicle security software 1110.

[0164] For example, the component 1101 may be a sensor. For example, the sensor may be used to obtain information regarding at least one of a state of the vehicle 1100 or a state surrounding the vehicle 1100. For example, the component 1101 may include a sensor 1410.

[0165] For example, component 1102 may be an electronic control unit (ECU), which may be used for engine control, transmission control, airbag control, and tire pressure management, for example.

[0166] For example, the component 1103 may be an instrument cluster. For example, the instrument cluster may refer to a panel of a dashboard that is located in front of the driver's seat. For example, the instrument cluster may be configured to display information required for driving to the driver (or passengers). For example, the instrument cluster may be used to display at least one of a visual element for indicating the revolutions per minute (RPM, revolutions per minute, or rotate per minute) of the engine, a visual element for indicating the speed of the vehicle 1100, a visual element for indicating the amount of remaining fuel, a visual element for indicating the status of the gear, or a visual element for indicating information obtained via the component 1101.

[0167] For example, the component 1104 may be a telematics device. For example, the telematics device may refer to a device that provides various mobile communication services such as location information, safe driving, etc. in the vehicle 1100 by combining wireless communication technology and global positioning system (GPS) technology. For example, the telematics device may be used to connect the vehicle 1100 with a driver, a cloud (e.g., security cloud 1106), and / or a surrounding environment. For example, the telematics device may be configured to support high bandwidth and low latency for 5G NR standard technology (e.g., 5G NR V2X technology). For example, the telematics device may be configured to support autonomous driving of the vehicle 1100.

[0168] For example, the gateway 1105 can be used to connect a network in the vehicle 1100 to a software management cloud 1109 and a security cloud 1106, which are networks outside the vehicle. For example, the software management cloud 1109 can be used to update or manage at least one software required for running and managing the vehicle 1100. For example, the software management cloud 1109 can work with an in-car security software 1110 installed in the vehicle. For example, the in-car security software 1110 can be used to provide a security function in the vehicle 1100. For example, the in-car security software 1110 can encrypt data transmitted and received via the in-car network using an encryption key obtained from an external authorized server for encryption of the in-car network. In various embodiments, the encryption key used by the in-car security software 1110 can be generated corresponding to vehicle identification information (vehicle license plate, vehicle identification number (VIN)) or information uniquely assigned to each user (user identification information, etc.).

[0169] In various embodiments, the gateway 1105 can transmit data encrypted by the in-vehicle security software 1110 based on the encryption key to the software management cloud 1109 and / or the security cloud 1106. The software management cloud 1109 and / or the security cloud 1106 can identify which vehicle or which user the data is received from by decrypting the data using a decryption key that can decrypt the data encrypted by the encryption key of the in-vehicle security software 1110. For example, since the decryption key is a unique key corresponding to the encryption key, the software management cloud 1109 and / or the security cloud 1106 can identify the sender of the data (e.g., a vehicle or a user) based on the data decrypted via the decryption key.

[0170] For example, the gateway 1105 may be configured to support the in-vehicle security software 1110 and may be associated with the control device 1000. For example, the gateway 1105 may be associated with the control device 1000 to support a connection between the client device 1107 connected to the security cloud 1106 and the control device 1000. In another example, the gateway 1105 may be associated with the control device 1000 to support a connection between the third party cloud 1108 connected to the security cloud 1106 and the control device 1000. However, this is not limiting.

[0171] In various embodiments, the gateway 1105 can be used to connect the vehicle 1100 with a software management cloud 1109 for managing the operating software of the vehicle 1100. For example, the software management cloud 1109 can monitor whether an update of the operating software of the vehicle 1100 is requested and, based on monitoring that an update of the operating software of the vehicle 1100 is requested, provide data for updating the operating software of the vehicle 1100 via the gateway 1105. As another example, the software management cloud 1109 can receive a user request for an update of the operating software of the vehicle 1100 from the vehicle 1100 via the gateway 1105 and, based on the receipt, provide data for updating the operating software of the vehicle 1100. However, this is not limiting.

[0172] 12 is a diagram illustrating the operation of an electronic device 101 for training a neural network based on a set of training data, according to one embodiment. The operations described with reference to FIG. 12 may be performed by the electronic devices of FIGS. 1a-11.

[0173] Referring to FIG. 12, in operation 1202, an electronic device according to an embodiment may obtain a set of training data. The electronic device may obtain a set of training data for supervised learning. The training data may include a pair of input data and ground truth data corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives the input data that is a pair of the ground truth data. The ground truth data may be obtained by the electronic device described with reference to FIGS. 1a to 12.

[0174] For example, when training a neural network to recognize images, the training data may include information about the images and one or more objects contained in the images. The information may include a category or class of objects identifiable via the images. The information may include a position, width, height, and / or size of a visual object in the image that corresponds to the object. The set of training data identified through operation 1202 may include multiple training data pairs. In the above example of training a neural network to recognize images, the set of training data identified by the electronic device may include multiple images and basis truth data corresponding to each of the multiple images.

[0175] 12, in operation 1204, an electronic device according to an embodiment may perform training of a neural network based on a set of training data. In an embodiment in which a neural network is trained based on supervised learning, the electronic device may input input data included in the training data to an input layer of the neural network. An example of a neural network including the input layer is described with reference to FIG. 13. From an output layer of the neural network that receives the input data via the input layer, the electronic device may obtain output data of the neural network that corresponds to the input data.

[0176] In one embodiment, the training of operation 1204 may be performed based on a difference between the output data and a base truth data included in the learning data and corresponding to the input data. For example, the electronic device may adjust one or more parameters (e.g., weights, described below with reference to FIG. 13) associated with the neural network so that the difference is reduced based on a gradient descent algorithm. The operation of the electronic device adjusting the one or more parameters may be referred to as tuning the neural network. The electronic device may perform the tuning of the neural network based on the output data using a function defined to evaluate the performance of the neural network, such as a cost function. The difference between the output data and the base truth data may be included as an example of a cost function.

[0177] 12, in operation 1206, an electronic device according to an embodiment may identify whether valid output data is output from the neural network trained by operation 1204. The output data being valid may mean that the difference (or cost function) between the output data and the basis truth data satisfies the conditions set for using the neural network. For example, if the average and / or maximum value of the difference between the output data and the basis truth data is less than or equal to a specified threshold, the electronic device may determine that valid output data is output from the neural network.

[0178] If the neural network does not output valid output data (ACT 1206-NO), the electronic device may repeatedly perform training of the neural network based on ACT 1204. The embodiment is not limited thereto, and the electronic device may repeatedly perform ACTs 1202, 1204.

[0179] Upon obtaining valid output data from the neural network (operation 1206-YES), the electronic device according to an embodiment may use the trained neural network based on operation 1208. For example, the electronic device may input other input data to the neural network as learning data, which is differentiated from the input data input to the neural network. The electronic device may use the output data obtained from the neural network that has received the other input data as a result of inference made on the other input data based on the neural network.

[0180] Figure 13 is a block diagram of an electronic device 101, according to one embodiment. The electronic device 101 of Figure 13 may include the electronic devices of Figures 1a-11. For example, the operations described with reference to Figure 12 may be performed by the electronic device 101 of Figure 13 and / or the processor 1310 of Figure 13.

[0181] 13, the processor 1310 of the electronic device 101 can execute computations related to the neural network 1330 stored in the memory 1320. The processor 1310 may include at least one of a center processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU). The NPU may be implemented as a chip separate from the CPU, or may be integrated into a chip such as the CPU in the form of a system on a chip (SoC). An NPU integrated into a CPU may be referred to as a neural core and / or an artificial intelligence (AI) accelerator.

[0182] Referring to FIG. 13, the processor 1310 may identify a neural network 1330 stored in the memory 1320. The neural network 1330 may include a combination of an input layer 1332, one or more hidden layers 1334 (or intermediate layers), and an output layer 1336. The aforementioned layers (e.g., the input layer 1332, the one or more hidden layers 1334, and the output layer 1336) may include multiple nodes. The number of hidden layers 1334 may vary depending on the embodiment, and a neural network 1330 including multiple hidden layers 1334 may be referred to as a deep neural network. The operation of training a deep neural network may be referred to as deep learning.

[0183] In one embodiment, when the neural network 1330 has a structure of a feed forward neural network, a first node included in a particular layer can be connected to all of the second nodes included in another layer before the particular layer. The parameters stored in the memory 1320 for the neural network 1330 can include weights assigned to the connections between the second node and the first node. In the neural network 1330 having the structure of a feed forward neural network, the value of the first node can correspond to a weighted sum of the values ​​assigned to the second node based on the weights assigned to the connections connecting the second node and the first node.

[0184] In one embodiment, when the neural network 1330 has the structure of a convolutional neural network, a first node included in a particular layer can correspond to a weighted weight for a portion of a second node included in another layer prior to the particular layer. Some of the second nodes corresponding to the first node may be identified by a filter corresponding to the particular layer. The parameters stored in the memory 1320 for the neural network 1330 can include weights representing the filters. The filter can include one or more of the second nodes used to calculate the weighted weight for the first node, and weights corresponding to each of the one or more nodes.

[0185] According to one embodiment, the processor 1310 of the electronic device 101 may perform training on the neural network 1330 using a training data set 1340 stored in the memory 1320. Based on the training data set 1340, the processor 1310 may perform the operations described with reference to FIG.

[0186] According to an embodiment, the processor 1310 of the electronic device 101 can perform object detection, object recognition, and / or object classification using the neural network 1330 trained based on the learning dataset 1340. The processor 1310 can input an image (or video) acquired via the camera 1350 to the input layer 1332 of the neural network 1330. Based on the input layer 1332 to which the image is input, the processor 1310 can sequentially obtain values ​​of nodes of the layers included in the neural network 1330 and obtain a set of values ​​of nodes of the output layer 1336 (e.g., output data). The output data can be used as a result of estimating information included in the image using the neural network 1330. The embodiment is not limited thereto, and the processor 1310 can input an image (or video) acquired from an external electronic device connected to the electronic device 101 to the neural network 1330 via the communication circuit 1360.

[0187] In one embodiment, the neural network 1330 trained to process an image can be used to identify regions in the image that correspond to objects (object detection) and / or to identify classes of objects represented in the image (object recognition and / or object classification). For example, the electronic device 101 can use the neural network 1330 to segment regions in the image that correspond to the objects based on a rectangular shape, such as a bounding box. For example, the electronic device 101 can use the neural network 1330 to identify at least one class of a plurality of specified classes that matches the object.

[0188] Figure 14 is a block diagram of an electronic device according to an embodiment. The operations of the electronic device described with reference to Figure 14 may be performed by the electronic device 101 and / or the processor 210 of Figures 1a-1b and 2.

[0189] 14, in an operation 1400, a processor of an electronic device according to an embodiment may capture an image. The image may be identified within a sequence of images (e.g., a video) captured successively by a camera. The image of operation 1400 may be referred to as an image frame.

[0190] 14, in operation 1402, according to one embodiment, a processor of the electronic device can identify an external object in the captured image. The external object can be identified based on object recognition. For example, the electronic device can identify the location and / or class of the external object in the image based on a neural network trained for object recognition.

[0191] 14, within operation 1404, according to one embodiment, a processor of the electronic device may classify external objects identified based on operation 1402 that may affect the operation of the vehicle. The processor may perform operation 1404 by comparing the likelihood for an external object identified from the image to a specified threshold. Based on operation 1404, the processor may identify at least one external object that may exceed a specified threshold.

[0192] 14, in operation 1406, a processor of an electronic device according to an embodiment may assign an object identification to each of the external objects classified by operation 1404. The object identification may include an identifier (or key value) that is uniquely assigned to the external object. The identifier may be assigned based on a natural number. The object identification may include at least one of a position, a class, and / or an identifier of the external object within the image of operation 1400.

[0193] 14, in operation 1408, a processor of an electronic device according to an embodiment may begin tracking an external object using an image frame-based linear interpolation method. For example, in operation 1410, a processor of an electronic device according to an embodiment may identify an external object having a particular identifier (e.g., a natural number i that gradually increases from 1) as an identifier in a first image frame.

[0194] 14, in operation 1412, according to one embodiment, a processor of the electronic device can store information about the external object identified based on operation 1410 and perform labeling. Based on the labeling, the processor can obtain labeling data for a particular identifier.

[0195] 14, in an operation 1414, the processor of the electronic device, according to an embodiment, can move to the mth image frame, which is spaced apart from the first image frame by the number of input image frames. In an operation 1416, the processor of the electronic device, according to an embodiment, can identify an external object having the natural number i of operation 1410 as an identifier in the mth image frame. In an operation 1418, the processor of the electronic device, according to an embodiment, can store information about the identified external object from the mth image frame and perform labeling.

[0196] 14, in operation 1420, according to an embodiment, the processor of the electronic device can use the external object information stored in the first image frame and the external object information stored in the mth image frame to store information about the external object information predicted to be present in the cth image frame and perform labeling, where c can be a natural number greater than 1 and less than m. In operation 1420, the external object information obtained for the cth image frame can correspond to an external object having the natural number i as an identifier.

[0197] 14, in operation 1422, according to one embodiment, the processor of the electronic device may complete the labeling operation of the external objects (e.g., external objects having a natural number i as an identifier) ​​of all the acquired image frames. For example, in operation 1424, the processor may identify whether the labeling operation for all the external objects present in the image frames has been completed. If the labeling operation for all the external objects has not been completed (1424-NO), the processor of the electronic device may perform operation 1426.

[0198] Referring to FIG. 14, in operation 1426, the processor of the electronic device, according to an embodiment, can identify other external objects. For example, based on the increase of the natural number i, the processor can identify other external objects that are distinguished from the external object having the natural number i as an identifier. Referring to FIG. 14, in operation 1428, the processor of the electronic device, according to an embodiment, can identify whether the identifier increased based on operation 1426 exceeds the number (n) of external objects identified based on operation 1402. If the identifier exceeds the number (1428-YES), the processor may interrupt the labeling operation. If the identifier is equal to or less than the number (1428-NO), the processor may perform tracking on the external object having the identifier increased based on operation 1426.

[0199] 15 is a diagram for explaining a process of obtaining labeling data related to a region selected by a user input in an image according to an embodiment of the present invention. In FIG. 15, reference numeral 1500 denotes an image frame for generating labeling data, and reference numerals 1505 and 1510 denote external objects present in the image frame 1500, respectively.

[0200] Reference numeral 1550 indicates that, in order to generate labeling data corresponding to the external objects 1505, 1510 according to an embodiment of the present invention, the electronic device displays bounding boxes 1555, 1560 corresponding to each external object 1505, 1510, respectively, and displays pixel values ​​of horizontal and vertical sizes of each bounding box 1555, 1560 according to a user input. According to an embodiment, the electronic device can identify the input to the bounding boxes 1555, 1560 based on the operations described with reference to FIG. 1b.

[0201] FIG. 16 is a diagram illustrating a process of generating labeling data for learning to identify an object in an image according to an embodiment of the present invention.

[0202] In order to generate labeling data for learning to identify objects in an image, a processor of an electronic device according to an embodiment of the present invention can manage an image file 1602 to be learned and a label file 1604 that has been labeled for that image file as one pair and store them in a storage device.

[0203] 16, "jpg" is used as the format of the image file 1602 to be learned, but this is merely one embodiment and does not limit the embodiment of the present invention. For example, various video compression formats such as "png" or "gif" may be used as the format of the image file 1602.

[0204] 16 shows that the extension of the label file 1604 for the image file 1602 is "json", but this is merely one embodiment for explaining a case where labeling information of objects in the image file 1602 is expressed as data structured by Javascript object grammar using JSON (Java Script Object Notation), and does not limit the embodiment of the present invention. For example, as the format of the labeling data, a data set format capable of expressing various hierarchical data structures such as XML (eXtensible Markup Language), csv (comma-separated values), COCO (Common Object in Context), VOC (Visual Object Classe), and TFRecord (Tensor Flow Record) can be used, and the labeling data format may change depending on the tool used for labeling.

[0205] Reference numeral 1650 in FIG. 16 indicates an example of an image file 1602 for which labeling data for object detection according to an embodiment of the present invention is generated. The image frame 1650 has a size of 1280 pixels horizontally and 650 pixels vertically, and there are a total of three objects for which labeling data is generated in the image frame 1650, including two pedestrian objects and one vehicle object. Specifically, reference numeral 1652 indicates a bounding box corresponding to pedestrian object 0, reference numeral 1654 indicates a bounding box for vehicle object 0, and reference numeral 1656 indicates a bounding box corresponding to pedestrian object 1. In FIG. 16, (X1, Y1) and (X2, Y2) are coordinate points on the X and Y axes indicating the position of the bounding box 1652 in the image frame 1650, and can respectively represent the vertex at the top left end and the vertex at the bottom right end of the bounding box 1652.

[0206] 16, (X3, Y3) and (X4, Y4) are coordinate points on the X and Y axes indicating the position of bounding box 1654 within image frame 1650, and can respectively represent the upper left vertex and lower right vertex of bounding box 1654. Additionally, (X5, Y5) and (X6, Y6) are coordinate points on the X and Y axes indicating the position of bounding box 1656 within image frame 1650, and can respectively represent the upper left vertex and lower right vertex of bounding box 1656.

[0207] Thus, in one embodiment of the present invention, in FIG. 16 it can be seen that the object class of the pedestrian object within bounding box 1652 is specified as “Pedestrian” and the object identifier for tracking that object is specified as “0”, the object class of the vehicle object within bounding box 1654 is specified as “car” and the object identifier for tracking that object is specified as “0”, and the object class of the pedestrian object in bounding box 1656 is specified as “pedestrian” and the object identifier for tracking that object is specified as “1”.

[0208] The object classes and object identifiers shown in FIG. 16 are merely examples, and may be set differently according to user specifications.

[0209] Fig. 17 is a diagram for explaining a hierarchical structure of labeling data generated according to an embodiment of the present invention. Referring to Fig. 17, reference numeral 1702 indicates an example of information on a tool used to generate labeling data in a target file, and reference numeral 1704 indicates an example of a path of a file corresponding to the labeling data. The tool information of reference numeral 1702 may include the type of tool used for labeling, version information of the tool, etc.

[0210] The tools for generating labeling data according to an embodiment of the present invention are software programs that can assign specific values ​​to training data before performing machine learning or deep learning modeling tasks, such as "labellmg", "CVAT (Computer Vision Annotation Tool), "LabelMe", "Labelbox", "VoTT", "imglab", "YOLO Mark", "OpenLabeling", "PixelAnnotationTool", "imagetagger", "Alturos.ImageAnnotation", "DeepLabel", "MedTagger", "Turktools", "Pixie", "OpenLabeler", "Anno-Mage", "CATMAID", "makesense.ai", "LOST (Label Object and Save The labeling data may include, for example, "Time", "annotorious", "sloth", and the like. However, this is merely an embodiment of the present invention and does not limit the embodiment of the present invention. Specifically, the tool for generating labeling data according to the embodiment of the present invention can be a software program that can specify various shapes of figures such as Rectangle, Polygon, Line, Point, etc. in an image for object detection and object identification learning, perform labeling, and store the bounding box related information as a data structure of a certain format. Also, reference numeral 1706 indicates an example of information on the width of a file (image file) corresponding to the labeling data, and reference numeral 1708 indicates an example of information on the height of a file (image file) corresponding to the labeling data.

[0211] Reference number 1750 is an example of flag information of data that is the target of labeling data. In the embodiment of the present invention, the data that is the target of labeling data is an image file, and since the object identification rate and object detection rate required for autonomous driving are greatly affected by the weather (cloudy, sunny, rainy, snowy, etc.) and the environment such as day and night, the flag information is set to sunny day, rainy day, and cloudy day, but this is not a limitation of the present invention.

[0212] In the embodiment of the present invention, the image frame 1650 that is the target of the labeling data is a sunny day, so in Fig. 17, among the flags included in the image flag 1750, the sunny day flag is assigned to "True", and the rainy day flag and the cloudy day flag are assigned to "False". That is, the data shown by reference number 1750 in Fig. 17 is merely one embodiment, and various flags that can be used to improve the object identification rate and object detection rate may exist.

[0213] 18 is a diagram for explaining a hierarchical structure of information related to external objects in labeling data generated by an embodiment of the present invention. Reference numeral 1810 in FIG. 18 is an example showing a labeling data structure of the bounding box 1652 corresponding to the pedestrian object 0 in FIG.

[0214] The Label parameter of reference number 1810 is assigned to "Pedestrian" because the object in the bounding box 1652 is a pedestrian, the Tracking ID parameter is assigned to "0", the Shape Type parameter is assigned to "Rectangle", and the Points parameter is assigned to at least two vertex coordinates (X1, Y1), (X2, Y2) of the bounding box 1652. In FIG. 18, the Label Flags parameter includes a total of three subparameters (interpolated, hidden, and clipped), and these three subparameters indicate whether the object included in the bounding box 1652 is an interpolated object, an object hidden by another object, or an object with a part clipped. Of the Label Flags of reference number 1810, the "interpolated" parameter is assigned to "True", and the "hidden" and "clipped" parameters are both assigned to "False".

[0215] The Label parameter of reference number 1830 is assigned to "Car" because the object in the bounding box 1654 is a vehicle, the Tracking ID parameter is assigned to "0", the Shape Type parameter is assigned to "Rectangle", and the Points parameter is assigned to at least two vertex coordinate values ​​((X3, Y3), (X4, Y4)) of the bounding box 1654. Also, among the Label Flags of reference number 1830, the "Interpolated", "Hidden", and "Cropped" parameters are all assigned to "False".

[0216] The Label parameter of reference numeral 1850 is assigned to "Pedestrian" since the object in the bounding box 1656 is a pedestrian, the Tracking ID parameter is assigned to "1" to distinguish it from other pedestrian objects in the bounding box 1656, the Shape Type parameter is assigned to "Rectangle", and the Points parameter is assigned to at least two vertex coordinate values ​​((X5, Y5), (X6, Y6)) of the bounding box 1656. And, among the Label Flags of reference numeral 1850, the "Interpolated", "Hidden", and "Cropped" parameters are all assigned to "False".

[0217] The information contained in the hierarchical structure of information regarding external objects in the labeling data shown in Figure 18 is illustrated to facilitate understanding of the present invention, and the external object classes may include external objects that need to be identified for autonomous driving of a vehicle, such as "Bus," "Truck," "Pick-Up Truck," "Sedan," "SUV (Sports Utility Vehicle)," "Bike," "Scooter," and "Motorcycle."

[0218] 18, the bounding box corresponding to the object to be labeled is assigned as a "Rectangle", so that Points representing its position coordinates are assigned as a pair of X-axis and Y-axis coordinate values, but when the Shape Type is "Polygon", pairs of X-axis and Y-axis coordinate values ​​of points generated by the user according to the appearance of the object to be labeled are saved according to the order of the points generated by the user. According to one embodiment of the present invention, when the Shape Type is "Polygon", an example of coordinates assigned to the Points parameter will be described with reference to FIG. 19 described later.

[0219] Fig. 19 is a diagram showing an example of coordinates assigned to the Points parameter when the Shape Type is "Polygon" according to an embodiment of the present invention. Referring to Fig. 19, when the "Shape Type" of the labeling data for a vehicle 1950, which is an external object in an image frame to be labeled, is "Polygon" and the user generates a total of 20 points from No. 1 to No. 20, pairs of X-axis coordinates and Y-axis coordinate values ​​from (X1, Y1) to (X20, Y20) corresponding to the total of 20 points from No. 1 to No. 20 are assigned, as shown in reference number 1970.

[0220] The labeling data described in this invention can be stored in the storage device of a user's local computer or in the storage device of a cloud server.

[0221] FIG. 20 is a diagram illustrating an example of dividing generated labeling data into a training set, a validation set, and / or a test set for efficient training of a neural network according to an embodiment of the present invention.

[0222] Referring to FIG. 20, an electronic device according to an embodiment of the present invention can analyze (2010) image flags of labeled data 2005 and classify the labeled data according to the analyzed image flags.

[0223] A processor of an electronic device according to an embodiment of the present invention can classify the labeled data 2005, such as reference number 2020, by weather-related flags according to the image flags or by time. Specifically, in FIG. 20, the labeled data 2010 can be classified into a sunny day, a cloudy day, a snowy day, a rainy day, and a foggy day according to the weather-related flags.

[0224] Further, the processor of the electronic device according to an embodiment of the present invention can classify the labeling data 2005 into morning, afternoon, night, sunset, sunrise based on a time flag corresponding to the image flag.

[0225] In addition, the processor of the electronic device according to an embodiment of the present invention may compare the amount of labeling data classified as shown by reference numeral 2020 with a value preset by a user, and determine a classification method for classifying the labeling data 2005 into a training set 2042, a validation set 2044, and / or a test set 2046 according to the comparison result. At this time, the preset value for determining the classification method of the labeling data is a value determined in advance by a user, and may be determined based on a prior experience value or an experimental value for determining whether labeling data is secured enough to generate the validation set 2044.

[0226] Reference numeral 2040 shows an example of a method in which the processor of the electronic device according to an embodiment of the present invention divides the labeling data 2005 into a training set 2042, a validation set 2044, and / or a test set 2046 when the amount of the labeling data 2005 is greater than a preset value. At this time, in reference numeral 2040, the processor of the electronic device can divide the labeling data 2005 into the training set 2042, the validation set 2044, and / or the test set 2046 according to a ratio preset by a user.

[0227] Meanwhile, reference numeral 2060 indicates an example of a manner in which the processor of the electronic device according to an embodiment of the present invention divides the labeling data 2005 into a training set 2042 and a test set 2046 when the amount of the labeling data 2005 is less than or equal to a preset value. At this time, reference numeral 2060 indicates that the processor of the electronic device divides the labeling data 2005 into the training set 2042 and the test set 2046 according to a ratio preset by a user, and uses K-fold cross validation, which is one of algorithms 2065 for verifying the reliability of a neural network, to verify the reliability of the training set 2042.

[0228] As mentioned above, according to one embodiment, an electronic device may include a memory and a processor. The processor may be configured to identify from the memory a first position associated with an external object in a plurality of images for a video and a first image of the plurality of images at a first timing. The processor may be configured to identify a second position associated with the external object in a second image of the plurality of images at a second timing after the first timing based on the first position in the first image. The processor may be configured to obtain one or more third positions associated with the external object corresponding to one or more third images included in a time interval between the first timing and the second timing based on the first position and the second position. The processor may be configured to store the first position, the one or more third positions, and the second position as labeling information indicative of a movement of the external object identified in the time interval of the video.

[0229] For example, the processor may be configured to obtain the one or more third positions by interpolating a first coordinate indicating the first position in the first image at the first timing and a second coordinate indicating the second position in the second image at the second timing using a length between the first timing and the second timing.

[0230] For example, the processor may be configured to obtain the one or more third positions by interpolating the first coordinate and the second coordinate based on timing within the time interval of the one or more third images.

[0231] For example, the processor may be configured to compare one or more feature points in a portion of the first image that includes the first location corresponding to the external object with one or more feature points in a second image that corresponds to the external object to identify the second location within the second image.

[0232] For example, the processor may be configured to modify the third position using at least one feature point included in the one or more third images based on identifying the second position relative to the second image at the second timing after a threshold interval after the first timing.

[0233] For example, the processor may be configured to input the first image and the second image into a model for recognizing the external object to identify the first position and the second position of an external object captured within the time interval.

[0234] For example, the electronic device may further include a display. The processor may be configured to display, on the display, a screen for playing the video. The processor may be configured to display, with any of the plurality of images being displayed on the screen based on an input indicating playback of the video, a visual object superimposed on the image displayed on the screen based on the labeling information and indicating a position of the external object.

[0235] For example, the processor may be configured to identify an input indicating a movement of the visual object within a state in which any of the one or more third images are displayed on the screen, and the processor may be configured to, based on the input, adjust a position of the external object corresponding to another of the one or more third images that is different from an image displayed on the screen based on a position of the visual object moved by the input.

[0236] As previously mentioned, a method of an electronic device according to an embodiment can include identifying, from a memory of the electronic device, a first position associated with an external object in a plurality of images for a video and a first image of the plurality of images at a first time. The method can include identifying, based on the first position in the first image, a second position associated with the external object in a second image of the plurality of images at a second time after the first time. The method can include obtaining, based on the first position and the second position, one or more third positions associated with the external object corresponding to one or more third images included in a time interval between the first time and the second time. The method can include storing the first position, the one or more third positions, and the second position as labeling information indicative of a movement of the external object identified within the time interval of the video.

[0237] For example, the obtaining operation may include an operation of obtaining the one or more third positions by interpolating a first coordinate indicating the first position in the first image at the first timing and a second coordinate indicating the second position in the second image at the second timing using a length between the first timing and the second timing.

[0238] For example, the obtaining operation may include obtaining the one or more third positions by interpolating the first coordinates and the second coordinates based on timing within the time interval of the one or more third images.

[0239] For example, identifying the second location may include identifying the second location in the second image by comparing one or more feature points included in a portion of the first image that includes the first location to a portion of the external object.

[0240] For example, the obtaining operation may include an operation of modifying the third position using at least one feature point included in the one or more third images based on identifying the second position relative to the second image at the second timing after a threshold interval after the first timing.

[0241] For example, the act of identifying the second location may include an act of inputting the first image and the second image into a model for recognizing the external object to identify the first location and the second location of an external object captured within the time interval.

[0242] For example, the method may include displaying, on a display of the electronic device, a screen for playing the video, and, with any of the plurality of images displayed within the screen based on an input indicating playback of the video, displaying, based on the labeling information, a visual object superimposed on the image displayed within the screen and indicating a location of the external object.

[0243] For example, the method may include an operation of identifying an input indicating a movement of the visual object within a state in which any of the one or more third images is displayed on the screen, and may further include an operation of adjusting, based on the input, a position of the external object corresponding to another one of the one or more third images different from the image displayed on the screen, based on a position of the visual object moved by the input.

[0244] As previously described, an electronic device according to an embodiment may include a display, a memory, and a processor. The processor may be configured to identify a first input indicating a selection of a first position associated with an external object within a first image of a video stored in the memory while displaying the first image on the display. The processor may be configured to perform a first type of computation for recognizing the external object based on the first input to identify a second position associated with the external object within a second image of a plurality of images for the video following a time interval starting from a timing of the first image. The processor may be configured to perform a second type of computation for interpolating the first position and the second position to obtain a third position associated with the external object within one or more third images included in the interval. The processor may be configured to display any of the first image, the one or more third images, and the second image on the display in response to a second input indicating playback of at least a portion of the video included in the time interval, and to display a visual object corresponding to any of the first position, the third position, or the second position, superimposed on the image displayed on the display.

[0245] For example, the processor may be configured to repeatedly perform the first type of calculation for recognizing the external object based on one or more feature points for each time interval from the timing of the first image.

[0246] For example, the processor may be configured to perform the second type of calculation to obtain the third position based on the first position, the second position, and timing of the one or more third images within the time interval.

[0247] For example, the processor may be configured to store the first location, the third location, and the second location in the memory as labeling information corresponding to the video.

[0248] For example, the processor may be configured to store data indicative of the timing of the first image in the labelling information in association with the first position.

[0249] As previously mentioned, a method of an electronic device according to one embodiment can include, while displaying a first image of a video stored in a memory of the electronic device on a display of the electronic device, identifying a first input indicating a selection of a first position associated with an external object in the first image. The method can include, based on the first input, performing a first type of calculation for recognizing the external object to identify a second position associated with the external object in a second image of a plurality of images for the video following a time interval starting from a timing of the first image. The method can include performing a second type of calculation for interpolating the first position and the second position to obtain a third position associated with the external object in one or more third images included in the time interval. The method can include, in response to a second input indicating playback of at least a portion of the video included in the time interval, displaying on the display any of the first image, the one or more third images, and the second image, and displaying a visual object corresponding to any of the first position, the third position, or the second position, superimposed on the image displayed on the display.

[0250] For example, the act of identifying the second location may include an act of repeatedly performing the first type of calculation for recognizing the external object based on one or more feature points for each of the time intervals from the timing of the first image.

[0251] For example, obtaining the third position may include performing the second type of calculation to obtain the third position based on the first position, the second position, and a timing of the one or more third images within the time interval.

[0252] For example, the method may further include storing, in the memory, the first location, the third location, and the second location as labeling information corresponding to the video.

[0253] For example, the storing operation may include storing data indicative of the timing of the first image in the labeling information in conjunction with the first position.

[0254] The aforementioned devices may be implemented with hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device may be described as being one, but a person skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations are possible, such as parallel processors.

[0255] The software may include a computer program, code, instructions, or a combination of one or more of these, and may configure or instruct a processing device to operate in a desired manner, either individually or collectively. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by a processing device or to provide instructions or data to a processing device. The software may be distributed over computer systems connected to a network, stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0256] The method according to the embodiment may be implemented in the form of program instructions executable via various computer means and recorded on a computer-readable medium. In this case, the medium may continue to store the computer-executable program or may temporarily store it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware combined, and may be distributed over a network, without being limited to a medium directly connected to any computer system. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROMs, RAMs, flash memories, and the like. Other examples of the medium include recording media managed by app stores that distribute applications, and sites, servers, and the like that supply or distribute various other software.

[0257] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art may make various modifications and variations from the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted by other components or equivalents, and still achieve suitable results.

[0258] Accordingly, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims

1. 1. An electronic device comprising: Memory, a processor, the processor comprising: Identifying from the memory a first location associated with an external object within a plurality of images for a video and a first image of the plurality of images at a first time; identifying a second location associated with the external object in a second image of the plurality of images at a second time subsequent to the first time based on the first location in the first image; Obtaining one or more third positions associated with the external object, corresponding to one or more third images included in a time interval between the first timing and the second timing, based on the first position and the second position; and The electronic device is configured to store the first position, the one or more third positions, and the second position as labeling information indicating movement of the external object identified within the time period of the video.

2. The processor, 2. The electronic device of claim 1, configured to obtain the one or more third positions by interpolating a first coordinate indicating the first position in the first image at the first timing and a second coordinate indicating the second position in the second image at the second timing using a length between the first timing and the second timing.

3. The processor, 3. The electronic device of claim 2, configured to obtain the one or more third positions by interpolating the first coordinates and the second coordinates based on timing within the time interval of the one or more third images.

4. The processor, 3. The electronic device of claim 2, configured to identify the second location in the second image by comparing one or more feature points included in a portion of the first image that includes the first location corresponding to the external object with one or more feature points included in the second image that corresponds to the external object.

5. The processor, 2. The electronic device of claim 1, configured to modify the third position using at least one feature point included in the one or more third images based on identifying the second position relative to the second image at the second timing after a threshold interval after the first timing.

6. The processor, 2. The electronic device of claim 1, configured to input the first image and the second image into a model for recognizing the external object to identify the first position and the second position of the external object captured within the time interval.

7. Further comprising a display; The processor, displaying a screen for playing the video on the display; 2. The electronic device of claim 1, wherein the electronic device is configured to display a visual object indicating the location of the external object, superimposed on the image displayed on the screen based on the labeling information, while any of the plurality of images is displayed on the screen based on an input indicating playback of the video.

8. The processor, identifying an input indicative of a movement of the visual object within the state of displaying any of the one or more third images on the screen; The electronic device of claim 7, configured to adjust, based on the input, a position of the external object corresponding to another image among the one or more third images that is different from the image displayed on the screen, based on a position of the visual object moved by the input.

9. 1. A method of an electronic device comprising: identifying, from a memory of the electronic device, a plurality of images for a video and a first image of the plurality of images at a first time, a first location associated with an external object; identifying a second location associated with the external object in a second image of the plurality of images at a second time subsequent to the first time based on the first location in the first image; obtaining, based on the first position and the second position, one or more third positions associated with the external object, corresponding to one or more third images included in a time interval between the first timing and the second timing; and The method includes an operation of storing the first position, the one or more third positions, and the second position as labeling information indicating movement of the external object identified within the time period of the video.

10. The obtaining operation includes:

10. The method of claim 9, comprising an operation of obtaining the one or more third positions by interpolating a first coordinate indicating the first position in the first image at the first timing and a second coordinate indicating the second position in the second image at the second timing using a length between the first timing and the second timing.

11. The obtaining operation includes:

11. The method of claim 10, comprising obtaining the one or more third positions by interpolating the first coordinate and the second coordinate based on timing within the time interval of the one or more third images.

12. The act of identifying the second location includes:

11. The method of claim 10, comprising identifying the second location in the second image by comparing one or more feature points included in a portion of the first image that includes the first location corresponding to the external object.

13. The obtaining operation includes:

10. The method of claim 9, further comprising: modifying the third position by using at least one feature point included in the one or more third images based on identifying the second position relative to the second image at the second time after a threshold interval after the first time.

14. The act of identifying the second location includes:

10. The method of claim 9, comprising inputting the first image and the second image into a model for recognizing the external object to identify the first position and the second position of the external object captured within the time interval.

15. displaying a screen for playing the video on a display of the electronic device; The method of claim 9, further comprising the operation of displaying a visual object indicating the location of the external object, superimposed on the image displayed on the screen based on the labeling information, while any of the plurality of images is displayed on the screen based on an input indicating playback of the video.

16. and identifying an input indicative of a movement of the visual object within the state of displaying any of the one or more third images on the screen. The method of claim 15, further comprising the operation of adjusting, based on the input, a position of the external object corresponding to another image among the one or more third images that is different from the image displayed on the screen, based on a position of the visual object moved by the input.

17. 1. An electronic device comprising: A display and Memory, a processor, the processor comprising: while displaying a first image of a video stored in the memory on the display, identifying a first input indicating a selection of a first location within the first image relative to an external object; performing a first type of computation for recognizing the external object based on the first input to identify a second location associated with the external object in a second image of a plurality of images for the video after a time interval starting from a timing of the first image; performing a second type of calculation to interpolate the first location and the second location to obtain a third location associated with the external object in one or more third images included in the interval; and displaying on the display any of the first image, the one or more third images, and the second image in response to a second input indicating playback of at least a portion of the video included in the time interval; An electronic device configured to display a visual object superimposed on an image displayed on the display and corresponding to either the first location, the third location, or the second location.

18. The processor, 18. The electronic device of claim 17, configured to repeatedly perform the first type of calculation for recognizing the external object based on one or more feature points for each time interval from the timing of the first image.

19. The processor, 20. The electronic device of claim 17, configured to perform the second type of calculation to obtain the third position based on the first position, the second position, and a timing of the one or more third images within the time interval.

20. The processor, The electronic device of claim 17 , configured to store in the memory the first location, the third location, and the second location as labeling information corresponding to the video.

Images