Electronic device for tracking external object in video and method thereof

Abstract

The invention relates to an electronic device for tracking an external object in a video and a method thereof. A processor of an electronic device according to an embodiment may identify a first position associated with an external object from within a first image at a first point in time among a plurality of images for a video. The processor may identify, based on the first location, a second location associated with the external object from within a second image at a second point in time after the first point in time in the plurality of images. The processor may acquire, based on the first position and the second position, one or more third positions associated with the external object and corresponding to one or more third images included within a time period between the first point in time and the second point in time. The processor may store the first position, the one or more third positions, and the second position as annotation information representing a motion of the external object.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a method thereof for tracking an external object in a video. Background Art

[0002] With the development of image object recognition technology, various forms of services are emerging. The above services can be applied to the fields of automatic driving, augmented reality, virtual reality, metaverse, etc., and can be provided through electronic devices owned by different users such as smartphones. These services may involve hardware and / or software mechanisms such as artificial intelligence that simulate human behavior and / or thinking. Technologies related to artificial intelligence may include neural network technology that uses simulated biological neural networks. Summary of the invention

[0003] Technical issues to be solved

[0004] For video playback, a scheme is being studied to more quickly identify, among images (eg, frames) included in the video, locations of external objects that are commonly captured in the images.

[0005] Technical Solution

[0006] An electronic device according to an embodiment may include a memory and a processor. The processor may identify a first position related to an external object from a first image at a first time point in a plurality of images for a video in the memory and in the plurality of images. The processor may identify a second position related to the external object from a second image at a second time point after the first time point in the plurality of images based on the first position in the first image. The processor may acquire one or more third positions related to the external object corresponding to one or more third images included in a time period between the first time point and the second time point based on the first position and the second position. The processor may store the first position, the one or more third positions, and the second position as annotation information representing the movement of the external object identified in the time period of the video.

[0007] According to a method of an electronic device of an embodiment, the method may include a step of identifying a first position related to an external object from a first image at a first time point in a plurality of images for a video in a memory of the electronic device and the plurality of images. The method may include a step of identifying a second position related to the external object from a second image at a second time point after the first time point in the plurality of images based on the first position in the first image. The method may include a step of acquiring one or more third positions related to the external object corresponding to one or more third images included in a time period between the first time point and the second time point based on the first position and the second position. The method may include a step of storing the first position, the one or more third positions, and the second position as annotation information representing the movement of the external object identified in the time period of the video.

[0008] According to an embodiment, an electronic device may include a display, a memory, and a processor. The processor may, in a state where the display displays a first image of a video stored in the memory, recognize a first input indicating a selection of a first position associated with an external object in the first image. The processor may perform a first type of calculation for identifying the external object based on the first input, so as to identify a second position associated with the external object in a second image after a time period starting from a time point of the first image from among multiple images of the video. The processor may perform a second type of calculation for interpolating the first position and the second position, so as to obtain a third position associated with the external object from one or more third images included in the time period. The processor may, in response to a second input indicating the playback of at least a portion of the video included in the time period, display the first image, one of the one or more third images, and the second image on the display, and superimpose a visual object corresponding to one of the first position, the third position, or the second position on the image displayed on the display.

[0009] According to the method of the electronic device of one embodiment as described above, the method may include the step of recognizing a first input indicating the selection of a first position related to an external object in a first image of a video stored in a memory of the electronic device in a state where the display of the electronic device displays a first image. The method may include the step of performing a first type of calculation for identifying the external object based on the first input, so as to identify a second position related to the external object in a second image after a time period starting from a time point of the first image from among multiple images of the video. The method may include the step of performing a second type of calculation for interpolating the first position and the second position, so as to obtain a third position related to the external object from one or more third images included in the time period. The method may include the step of displaying the first image, one of the one or more third images and the second image on the display in response to a second input indicating the playback of at least a portion of the video included in the time period, and superimposing and displaying a visual object corresponding to one of the first position, the third position or the second position on the image displayed on the display.

[0010] Beneficial Effects

[0011] According to an embodiment, an electronic device may obtain annotation information for model training for object recognition from a video including a plurality of images recorded sequentially.

[0012] According to an electronic device of an embodiment, it is possible to more quickly obtain annotation information of the positions of visual objects contained in each of a plurality of images contained in a video. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1a to Figure 1b An operation example of an electronic device acquiring annotation information according to an embodiment is shown.

[0014] Figure 2 A block diagram example of an electronic device according to an embodiment is shown.

[0015] Figure 3 An example of an operation flow chart performed by an electronic device according to an embodiment is shown.

[0016] Figure 4 An operation example of an electronic device recognizing an object from an image included in a video according to an embodiment is shown.

[0017] Figure 5a to Figure 5b An example of an electronic device operation of acquiring annotation information of another image based on annotation information of two images included in a video is shown.

[0018] Figure 6 An example of an operation flow chart performed by an electronic device according to an embodiment is shown.

[0019] Figure 7 An example of an interpolation operation performed by an electronic device according to an embodiment is shown.

[0020] Figure 8 An example block diagram of an autonomous driving system for a vehicle according to an embodiment is shown.

[0021] Fig. 9 and Fig.10 An example block diagram of an autonomous driving mobile body according to an embodiment is shown.

[0022] Fig.11 An example of a gateway associated with a user equipment according to an embodiment is shown.

[0023] Fig.12 is a schematic diagram for explaining the operation of an electronic device for training a neural network based on a learning data set.

[0024] Fig.13 is a block diagram of an electronic device according to an embodiment.

[0025] Fig.14 is an example of an operation flow chart performed by an electronic device according to an embodiment.

[0026] Fig.15 is a schematic diagram for illustrating a process of acquiring annotation data related to a region selected according to a user input in an image according to an embodiment of the present invention.

[0027] Fig.16 is a schematic diagram for illustrating a process of generating annotation data for object recognition learning from an image according to an embodiment of the present invention.

[0028] Fig.17 is a schematic diagram for illustrating a hierarchical structure of generated annotation data according to an embodiment of the present invention.

[0029] Fig.18 is a schematic diagram for illustrating a hierarchical structure of external object information in generated annotation data according to an embodiment of the present invention.

[0030] Fig.19 is a diagram of an example of coordinates assigned to Points Parameter when Shape Type is "Polygon" according to an embodiment of the present invention.

[0031] Fig. 20is a diagram for illustrating an example of dividing the generated labeled data into a training set, a validation set and / or a test set in order to achieve effective training of a neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0033] The various embodiments of the present disclosure and the terms used therein are not intended to limit the specific implementation of the technology described herein, but should be understood to include various changes, equivalents and / or substitutes of the relevant embodiments. With respect to the description of the accompanying drawings, similar constituent elements may use similar reference symbols. Unless the context has a clear different meaning, a singular expression may include a plural expression. In this article, expressions such as "A or B", "at least one of A and / or B", "A, B or C" or "at least one of A, B and / or C" may include all possible combinations of the enumerated items. Expressions such as "first", "second", "first" or "second" may modify the relevant constituent elements regardless of their order or importance, and are only used to distinguish one constituent element from another constituent element, and are not used to limit the relevant constituent elements. When referring to a certain (e.g., first) constituent element "(functionally or communicatively) connected" or "connected to" another (e.g., second) constituent element, this means that the certain constituent element may be directly connected to the other constituent element, or connected through other constituent elements (e.g., third constituent element).

[0034] The term "module" as used herein includes a unit composed of hardware, software or firmware, and can be used interchangeably with terms such as logic, logic block, component or circuit. A module can be an integral component, or a minimum unit or part thereof that performs one or more functions. For example, a module can be composed of an ASIC (application-specific integrated circuit).

[0035] Figure 1a to Figure 1b FIG. 2 shows an example of an operation of an electronic device acquiring annotation information according to an embodiment. Figure 1a to Figure 1b, exemplarily showing some form factors of the electronic device 101. In terms of user ownership, the electronic device 101 may be referred to as a terminal, a user device and / or a user terminal. The electronic device 101 may include a personal computer (PC), such as a desktop 101-1 and / or a laptop 101-2. The electronic device 101 may include a smartphone 101-3, a smart pad and / or a tablet computer. The terminal may also include smart accessories such as a smart watch and / or a head-mounted device (HMD, head-mounted device).

[0036] Reference Figure 1a According to an embodiment, the electronic device 101 may display a screen 110. Using the screen 110, the electronic device 101 may visualize information (eg, video 120). In order to display the screen 110 and / or output the information, the reference Figure 2 One or more hardware included in the electronic device 101 is exemplarily described. Hereinafter, the screen 110 may refer to a user interface (UI) displayed in at least a portion of the display. For example, the screen may include a window of a Windows operating system and / or an activity of an Android operating system. The embodiment is not limited thereto, and the screen 110 may be formed in the external space by light output from the electronic device 101 to the external space. For example, the screen 110 may be formed on a plane in the external space on which the light is projected. For another example, the screen 110 may be displayed in three dimensions in the form of a hologram in the external space.

[0037] According to one embodiment, the electronic device 101 may display a screen 110 for playing a video 120. The video 120 may include a set of images displayed sequentially according to a frame rate (or frames per second (fps)). The images contained in the image set may be referred to as frames, frame data, and / or frame images. The video 120 may include an audio signal output during the sequential display of the images. In terms of including visual information (e.g., the image set) and auditory information (e.g., the audio signal), the video 120 may be referred to as multimedia content (or media content).

[0038] Reference Figure 1a, exemplarily showing multiple images 120-1, 120-k, 120-N recognized by the electronic device 101 from the video 120. In the state of playing the video 120, it is assumed that the electronic device 101 is Figure 1a Among the multiple images shown, one image is displayed before the other images (eg, the kth image 120-k) Figure 1a The image shown on the left (e.g., the first image 120-1). For example, in the state, the k-th image 120-k may be displayed by the electronic device 101 after the first image 120-1. For example, in the state, the N-th image 120-N may be displayed by the electronic device 101 after the k-th image 120-k. For example, the electronic device 101 may display the first image 120-1, the k-th image 120-k, and the N-th image 120-N in this order.

[0039] According to one embodiment, the electronic device 101 can identify one or more external objects from the video 120. According to one embodiment, the external object is an object existing around the system (e.g., vehicle) in which the electronic device 101 is installed, including pedestrians, vehicles, bicycles, personal mobility devices (PM, Personal Mobility), road signs, lane markings, etc. The process of identifying the one or more external objects may include the operation of generating and / or obtaining information about the one or more external objects captured by the video 120 from the video 120. The information may include data representing a part related to the one or more external objects in at least one of the multiple images of the video 120. The information may include data representing the classification (classification or category) of the one or more external objects. In one embodiment, the process of the electronic device 101 identifying the one or more external objects may include the operation of generating and / or obtaining information representing the location related to the one or more external objects from at least one of the multiple images contained in the video 120. According to an embodiment, the electronic device 101 may obtain information including recognition results of the one or more external objects based on recognizing the one or more external objects from the video 120. The information obtained by the electronic device 101 may be referred to as metadata and / or labeling information corresponding to the video 120.

[0040] According to one embodiment, the information of one or more external objects related to the video 120 acquired by the electronic device 101 can be called annotation information from the perspective of training the external object recognition model. For example, the annotation information can be used for supervised learning of the model together with the video 120 matching the annotation information. The model is a recognition model implemented by software or hardware that uses a large number of artificial neurons (or nodes, perceptrons) to simulate the computing power of biological systems. Based on the model, the electronic device 101 can perform operations similar to human cognitive functions or learning processes. The supervised learning of the model can include using input data (e.g., video 120), the model output data for the input data, and ground truth data (e.g., annotation information corresponding to the video 120) to modify the weights assigned to multiple nodes and / or connections between the nodes included in the model. The model trained by the video 120 and the annotation information corresponding to the video 120 can output the result of identifying the external object from the other video when receiving other videos different from the video 120. The information of the one or more external objects may be used for purposes other than supervised learning. For example, from the perspective of tracking and / or monitoring external objects contained in the video 120, the information of the one or more external objects associated with the video 120 may be referred to as object tracking information.

[0041] Reference Figure 1a, exemplarily showing a screen 110 displayed by an electronic device 101 in order to generate annotation information corresponding to a video 120. The electronic device 101 can display a first image 120-1 of the video 120 at a first time point. The electronic device 101 can display a panel 112 containing one or more visual objects for controlling the video 120 together with the first image 120-1 in the screen 110. A visual object may refer to an object that can be deployed in the screen 110 for information transmission and / or interaction, such as text, images, icons, videos, buttons, check boxes, radio buttons, text boxes, sliders and / or tables. A visual object may be referred to as a visual element, a UI element, a view object and / or a view element. While displaying the first image 120-1 in the video 120, the electronic device 101 can recognize user input for selecting an area in the first image 120-1 related to an external object. Referring to Figure 1b , illustrating exemplary operations of the electronic device 101 for recognizing user input.

[0042] exist Figure 1a In an exemplary case, in the first image 120-1, the visual object 130-1 represents a vehicle, and the electronic device 101 displaying the first image 120-1 including the visual object 130-1 can recognize a user input for selecting and / or segmenting the visual object 130-1. For example, the electronic device 101 can recognize an input for selecting apexes A1, B1, C1, and D1 of the region 140-1 of the first image 120-1 including the visual object 130-1.

[0043] In one embodiment, in response to an input representing a selection of vertices A1, B1, C1, D1, the electronic device 101 may display a visual object in a shape of a bounding box representing the area 140-1 corresponding to the input. For example, the vertices of the bounding box may match the vertices A1, B1, C1, D1 of the area 140-1. The electronic device 101 may connect the vertices A1, B1, C1, D1 with a line of a specified color to display the bounding box. The specified color of the line used to display the bounding box may be related to the result of identifying the visual object 130-1 contained in the area 140-1. Figure 1aIn the exemplary case of , based on recognizing that the external object corresponding to the visual object 130-1 is a vehicle, the electronic device 101 may set the designated color of the bounding box to a designated color corresponding to the vehicle. Embodiments are not limited thereto, and the electronic device 101 may display overlapping text (e.g., "vehicle") near or on the bounding box to display the result of the external object recognized in the area 140-1 of the first image 120-1 contained in the bounding box.

[0044] According to one embodiment, the electronic device 101 may identify one or more locations related to the external object from other images different from the first image 120-1 (e.g., the kth image 120-k and / or the Nth image 120-N) based on identifying a location related to the external object in the first image 120-1 (e.g., the area 140-1). In one embodiment, the electronic device 101 may identify a location related to the external object from other images different from the first image 120-1 based on a user input for selecting a location related to the external object in the first image 120-1 (e.g., the location of the area 140-1). The electronic device 101 may identify a location related to the external object from other images different from the first image 120-1 without requiring other user inputs different from the user input corresponding to the first image 120-1. The operation of the electronic device 101 to identify the location of the external object within other images in the video 120 based on the user input to the first image 120-1 will refer to Figure 3 and / or Figure 6 Provide explanation.

[0045] According to an embodiment, the electronic device 101 may identify an external object (in the second image at a second time point after the first time point of the first image 120-1) related to the region 140-1 in the video 120 based on the region 140-1 identified by the input related to the first image 120-1. Figure 1a In order to identify the second location, the electronic device 101 may compare feature information (e.g., one or more feature points) of each of the first image 120-1 and the second image. In order to identify the second location, the electronic device 101 may use a model trained for object recognition. The operation of the electronic device 101 to identify the second location based on the feature information will be referred to as Figure 4 Provide explanation.

[0046] As an example without limiting the embodiment, extracting feature information of all other images different from the first image 120-1 from the multiple images (image frames) contained in the video 120, or inputting all the other images into the model, may result in an increase in the amount of calculation and / or power consumption. This will cause a large load on the electronic device 101 according to an embodiment of the present invention, thereby negatively affecting the effective operation of the driving assistance system or the vehicle automatic driving system for assisting the driver of a moving vehicle in a road environment where very variable and unpredictable conditions frequently occur, especially in an urban environment. According to an embodiment, the electronic device 101 can change the computing method used to identify external objects in the process of identifying external objects from other images based on the area 140-1 of the first image 120-1 selected by the user. The electronic device 101 can change the computing method applicable to each of the other images in different computing methods based on the respective time points of the other images in the video 120. The calculation method may include a first type of calculation method for recognizing an external object based on feature points of an image, and a second type of calculation method related to interpolation of positions of a specific external object recognized in different images.

[0047] Reference Figure 1a , based on an input representing a position (e.g., region 140-1) in a first image 120-1 related to an external object, the electronic device 101 may identify a position (e.g., region 140-N) related to the external object in an Nth (Nth) image (or an image at an Nth time point) acquired after a first time point of the first image 120-1, i.e., an Nth image 120-N. For example, the electronic device 101 may compare one or more feature points contained in region 140-1 of the first image 120-1 with feature points of the Nth image 120-N to identify region 140-N related to the external object in the Nth image 120-N. When the screen 110 displays the Nth image 120-N, the electronic device 101 may display a visual object having a bounding box form corresponding to region 140-N in an overlaid manner with the Nth image 120-N. For example, the electronic device 101 may display a bounding box in the form of lines connecting vertices AN, BN, CN, and DN of region 140-N.

[0048] Reference Figure 1a, when the region 140-N of the Nth image 120-N is identified based on the feature points in the region 140-1 of the first image 120-1, the electronic device 101 can identify the position related to the external object corresponding to the regions 140-1 and 140-N in one or more images included in the time period of the video 120 divided by the first image 120-1 and the Nth image 120-N. Figure 1a , the time period of the video 120 divided by the first image 120-1 and the N-th image 120-N may be started at the first time point of the first image 120-1 (beginning timing) and ended at the N-th time point of the N-th image 120-N (endtiming). In one embodiment, the electronic device 101 may identify a region 140-k related to the k-th image 120-k (where "k" is a natural number greater than 1 and less than N) contained in the time period based on interpolation of the region 140-1 of the first image 120-1 and the region 140-N of the N-th image 120-N.

[0049] In one embodiment, in a state where region 140-k is identified based on the interpolation of region 140-1 and region 140-N, the electronic device 101 may determine the coordinates of vertices Ak, Bk, Ck, and Dk of region 140-k in the kth image 120-k based on the coordinates of vertices A1, B1, C1, and D1 of region 140-1 and the coordinates of vertices AN, BN, CN, and DN of region 140-N. For example, the coordinates of vertex Ak of ​​region 140-k may be the coordinates of an internally dividing point on a line segment connecting vertex A1 of region 140-1 and vertex AN of region 140-N. The proportion of the line segment divided by the internal dividing point may be related to a time point of the kth image 120-k within the time period. An example of operation of the electronic device 101 for identifying the region 140 - k and / or the vertices of the region 140 - k of the k-th image 120 - k based on the interpolation of the vertices of the regions 140 - 1 and 140 -N will be described with reference to Figure 5a to Figure 5b Provide explanation.

[0050] Reference Figure 1a, although the operation of the electronic device 101 identifying the regions 140-1, 140-k and 140-N related to the external object in the first image 120-1, the kth image 120-k and the Nth image 120-N is described, the embodiment is not limited thereto. For example, the electronic device 101 may identify the regions related to the external object in all images between the first image and the Nth image including the kth image based on the interpolation of the regions 140-1 and 140-N. Although not shown in the figure, for other images after the Nth image in the video 120, the electronic device 101 may identify the regions related to the external object based on feature points and / or interpolation. According to one embodiment, the electronic device 101 may store the result of identifying the regions related to the external object from the multiple images included in the video 120 as annotation information corresponding to the video 120.

[0051] According to an embodiment, the electronic device 101 can visualize the results of the recognition from the multiple images of the video 120 for the area related to the external object in the screen 110. Figure 7 As described above, the electronic device 101 may recognize a user input for adjusting (or calibrating) the recognition result of the area.

[0052] Reference Figure 1b , which illustrates an exemplary operation of the electronic device 101 recognizing an input representing a selection of an external object in a specific image (eg, the first image 120-1) of the video 120. Figure 1b According to an embodiment, the electronic device 101 may display a screen (e.g., screens 150-1 and 150-2) for acquiring annotation data corresponding to the first image 120-1. The electronic device 101 may display a video 120-1 based on an input indicating selection of the first image 120-1 from among the multiple images included in the video 120. Figure 1b Either of the middle pictures 150-1 or 150-2.

[0053] Reference Figure 1b, based on the screen 150-1 corresponding to the first image 120-1 of the video 120, the electronic device 101 can recognize an input indicating the selection of the rectangular area 160-1. The input indicating the selection of the area 160-1 may include a drag input (drag input) from the vertex A1 to the vertex A3 of the area 160-1, which is recognized based on the mouse connected to the electronic device 101. The embodiment is not limited to this, and the electronic device 101 can also recognize the input indicating the selection of the area 160-1 based on a drag gesture performed on the touch screen. The electronic device 101 that recognizes the input can obtain the category (class or category) of the object captured by the area 160-1. For example, the electronic device 101 can display a pop-up window to receive text representing the category. Based on the text received from the user through the pop-up window, the electronic device 101 can identify the category (class or category) of the object (e.g., the visual object 130-1) captured in the area 160-1. The electronic device 101 may store information about the area 160 - 1 (eg, coordinates, categories, and / or classifications of the vertices A1 , A2 , A3 , A4 ) as annotation data corresponding to the first image 120 - 1 .

[0054] In one embodiment, in order to obtain the annotation data, the shape of the area formed in the first image 120-1 is not limited to a rectangle (eg, Figure 1b 160-1 in FIG. Figure 1b The screen 150-2 of the video 120 can recognize an input indicating that a polygonal area 160-2 is selected from the first image 120-1 of the video 120. The input indicating that the area 160-2 is selected can be recognized by a gesture of clicking and / or touching the vertices B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, and B14 of the polygon in sequence. Figure 1b As shown in the screen 150-2, the electronic device 101 that recognizes the input can display the polygonal area 160-2 formed in the first image 120-1 by connecting the vertices. The electronic device 101 can recognize the input representing the selection area 160-2, and can also recognize the input for setting the category (class or category) of the visual object 130-1 in the first image 120-1 divided by the area 160-2. The electronic device 101 can store the coordinates of the vertices B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14 of the area 160-2 and the combination of the category (class or category) as the annotation data of the first image 120-1.

[0055] As described above, according to one embodiment, the electronic device 101 can identify the external object in other images different from the specific image based on an input indicating the selection of the external object in a specific image (e.g., the first image 120-1) among the multiple images included in the video 120. When the electronic device 101 identifies the external object, it can be performed according to a combination of a first type of calculation method based on feature points and / or models that requires a relatively large amount of calculation and a second type of calculation method based on interpolation that requires a relatively small amount of calculation. For example, the electronic device 101 can perform an operation of identifying an external object in all multiple images of the video 120 without identifying the external object in other images different from the specific image. Since the electronic device 101 alternately applies calculation methods that require different amounts of calculation on different images in the video 120, the amount of calculation required to obtain the annotation information corresponding to the video 120 can be reduced. Since the amount of calculation required to obtain the annotation information is reduced, the electronic device 101 can obtain the annotation information corresponding to the video 120 more quickly.

[0056] Below, we will refer to Figure 2 Description Used to perform reference Figure 1a to Figure 1b An exemplary structure of the electronic device 101 for illustrating operation.

[0057] Figure 2 A block diagram example of an electronic device 101 according to an embodiment is shown. Figure 2 The electronic device 101 may include Figure 1a to Figure 1b The electronic device 101 in. Figure 2 The electronic device 101 in the embodiment may include at least one of a processor 210, a memory 220, or a display 230. The processor 210, the memory 220, and the display 230 may be electrically connected and / or operably coupled with each other through electronic components such as a communication bus 202. Hereinafter, hardware operably coupled means that a direct connection or an indirect connection between hardware is established by a wired or wireless manner so that a second hardware is controlled by a first hardware in the hardware. Although illustrated based on different modules, the embodiments are not limited thereto. Figure 2 Some of the hardware in the electronic device 101 (for example, at least a portion of the processor 210 and the memory 220) may be included in a single integrated circuit such as a SoC (system on a chip). The type and / or quantity of hardware included in the electronic device 101 is not limited to Figure 2 For example, the electronic device 101 may only include Figure 2 Some of the hardware elements shown.

[0058] According to one embodiment, the processor 210 of the electronic device 101 may include hardware for processing data based on more than one instruction. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU) and / or an application processor (AP). The number of processors 210 may be more than one. For example, the processor 210 may have a structure of a multi-core processor such as a dual core, a quad core or a hexa core.

[0059] According to one embodiment, the memory 220 of the electronic device 101 may include a hardware element for storing data and / or instructions input to and / or output from the processor 210. The memory 220, for example, may include a volatile memory such as a RAM (random-access memory) and / or a non-volatile memory such as a ROM (read-only memory). The volatile memory, for example, may include at least one of a DRAM (dynamic RAM), a SRAM (static RAM), a Cache RAM, and a PSRAM (pseudo SRAM). The non-volatile memory, for example, may include at least one of a PROM (programmable ROM), an EPROM (erasable PROM), an EEPROM (electrically erasable PROM), a flash memory, a hard disk, an optical disk, an SSD (solid state drive), and an eMMC (embedded multi media card).

[0060] According to one embodiment, in the memory 220 of the electronic device 101, one or more instructions (or commands) for the processor 210 to perform operations and / or operations on data may be stored. The set of one or more instructions may include firmware, operating system, process, routine, subroutine and / or application. For example, when a plurality of instruction sets (set of a plurality of instructions) distributed in the form of operating system, firmware, driver and / or application are executed, the electronic device 101 and / or the processor 210 may execute Figure 3 and / or Figure 6 At least one operation in. Hereinafter, an application is installed in the electronic device 101 means that one or more instructions provided in the form of an application are stored in the memory 220 of the electronic device 101, and the one or more applications are stored in a format executable by the processor 210 of the electronic device 101 (for example, a file with an extension specified by the operating system of the electronic device 101).

[0061] According to an embodiment, the display 230 of the electronic device 101 may output visual information (eg, Figure 1a ). For example, the display 230 may be controlled by a controller such as a GPU (graphic processing unit) to output visualized information to the user. The display 230 may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LED). The LED may include an organic light emitting diode (OLED). The display 230 may include a flat panel display (FPD) and / or electronic paper. The embodiment is not limited thereto, and the display 230 may have a shape that is at least partially curved or a deformable shape.

[0062] According to an embodiment, the electronic device 101 may obtain the annotation information 240 from the video 120 stored in the memory 220 . Figure 2 The video 120 may include Figure 1a to Figure 1b For example, the electronic device 101 can identify from the video 120 a plurality of images (eg, Figure 1a to Figure 1bThe annotation information 240 acquired by the electronic device 101 from the video 120 may include the location of the external object associated with each of the plurality of images. According to an embodiment, the electronic device 101 may obtain the annotation information 240 from the video 120 without user input for all images (e.g., referring to Figure 1b In the case of an input of a description, based on the positions of external objects associated with some (several) images, labeling information 240 of all the images is obtained.

[0063] According to one embodiment, the electronic device 101 may obtain positions corresponding to other images between the two images by interpolating positions corresponding to two images spaced apart from each other in an image sequence included in the video 120. The position of the external object associated with the two images spaced apart from each other may be identified by a user of the electronic device 101 and / or a model included in the electronic device 101 for feature point analysis. When the time period between the two images spaced apart from each other is relatively long (for example, exceeding a specified threshold length), the electronic device 101 may calibrate the position obtained by the interpolation based on feature points of more than one image within the time period. According to one embodiment, the annotation information 240 stored in the memory 220 by the electronic device 101 may include data representing a position associated with the external object in each of the images. The data may include Figure 1a At least one of the coordinates, width, height, aspect ratio, or size of at least one of the vertices of the middle region 140-1, 140-k, 140-N.

[0064] According to one embodiment, the annotation information 240 acquired by the electronic device 101 may represent the movement of the external object captured in the video 120. For example, the annotation information 240 may include the position of the area connected to the external object in each of the multiple images contained in the video 120 during the time period in which the multiple images contained in the video 120 are played in sequence. The electronic device 101 may identify the result of identifying the external object in each of the multiple images of the video 120 based on the annotation information 240. The result may include the position of the area related to the external object in a specific image. The electronic device 101 may use the multiple images and the annotation information 240 to adjust (or train) a model (or neural network) for identifying the external object. In order to improve the performance of the model, it may be necessary to adjust the model based on a relatively large number of images. According to one embodiment, the electronic device 101 may generate the annotation information 240 of all the multiple images contained in the video 120 using the information of the external object related to a specific image of the video 120 (for example, information representing the area related to the external object in the specific image). Since the electronic device 101 can generate the annotation information 240 of all the multiple images, the electronic device 101 can more quickly obtain the required number of images for adjusting the model.

[0065] Below, we will refer to Figure 3 ,illustrate Figure 2 An example of operations performed by the electronic device 101 in FIG.

[0066] Figure 3 An example of an operation flow chart performed by an electronic device according to an embodiment is shown. Figure 1a to Figure 1b , Figure 2 The electronic device 101 and / or the processor 210 may execute the reference Figure 3 Instructions for operation.

[0067] Reference Figure 3 In operation 310, according to one embodiment, a processor of the electronic device may identify a plurality of images for a video. The video in operation 310 may include Figure 1a to Figure 2 The processor can identify a plurality of images for playing the video from a file having a format related to MPEG (motion picture experts group). The file can be stored in a memory (e.g., Figure 2 The processor may identify the plurality of images and the order in which the plurality of images are played based on operation 310.

[0068] Reference Figure 3In operation 320, a processor of an electronic device according to an embodiment may identify a first position associated with an external object in a first image at a first time point among a plurality of images. The processor may display a display (e.g., Figure 2 In a state where a first image is displayed on a display 230 in the display 230, a first input indicating selection of the first position is recognized (for example, referring to Figure 1b The first image in operation 320 may include Figure 1a For example, Figure 1a The screen 110 of FIG. 110 may be displayed by the processor that recognizes the input indicating the selection of the area 140-1 of the first image 120-1. Figure 1a Based on the input of the region 140-1 in the first image, the processor may identify a first position (e.g., the position of the region 140-1) related to the external object in the first image 120-1. In one embodiment, the first position identified by the processor based on operation 320 may include a visual object (e.g., Figure 1a The at least one coordinate may include Figure 1a In the region 140 - 1 divided by vertices A1 , B1 , C1 , and D1 in the image 120 - 1 .

[0069] Reference Figure 3 In operation 330, the processor of the electronic device according to an embodiment may identify, based on the first position of operation 320, a second position related to an external object in a second image at a second time point after a first time point in a plurality of images. In a state where the first input for identifying the first position in operation 320 is identified, the processor may perform a first type of calculation for identifying an external object based on the first input, thereby identifying the second position in the second image after a time period starting from the first time point. In the plurality of image sequences for playing a video in operation 310, one or more third images may be arranged between the first image at the first time point and the second image at the second time point.

[0070] In one embodiment, the first type of calculation performed by the processor to identify the second location in operation 330 may be related to feature points of the first image and the second image. For example, the processor may extract at least one feature point from the first image that is associated with the first location of the first image identified in operation 320. The processor may extract one or more feature points from the second image of operation 330. The processor may compare the at least one feature point extracted from the first image with the one or more feature points extracted from the second image. Based on the comparison, the processor may identify an area in the second image that is similar to an area in the first image that contains the first location (e.g., Figure 1a 140-1) in a similar region (e.g., Figure 1a The second location in operation 330 may be contained within the region of the second image identified by the processor.

[0071] Embodiments are not limited thereto, and the first type of calculation performed by the processor in order to identify the second position in operation 330 may be associated with a training model for identifying external objects. For example, the processor may input data of an external object associated with the first position of the first image and the second image into the model. Based on data output by the model to which the data and the second image are input, the processor may identify a second position associated with the external object in the second image.

[0072] Reference Figure 3 In operation 340, a processor of an electronic device according to an embodiment may obtain, based on the first position and the second position, one or more third positions related to an external object corresponding to one or more third images contained in a time period between a first time point and a second time point. For example, the processor may perform a second type of calculation for interpolating the first position of operation 320 and the second position of operation 330 to obtain the one or more third positions. The one or more third positions obtained by the processor may be arranged on a line connecting the first position and the second position. The one or more third positions on the line may be related to the time points of the one or more third images. The processor may obtain one or more third positions in operation 340 based on a calculation amount that is less than the calculation amount performed in operation 330 to extract feature points.

[0073] Reference Figure 3 In operation 350, the processor of the electronic device according to an embodiment may store the first position, one or more third positions, and the second position as annotation information representing the movement of the external object recognized within the time period of the video. Figure 2 The annotation information 240 in operation 350 may include the annotation information in operation 350. According to a processor of one embodiment, parameters representing images corresponding to the positions identified by operations 320 to 340 in multiple images may be stored in the annotation information. For example, the annotation information stored by the processor may include parameters representing that the first position corresponds to the first image (e.g., a timestamp associated with data representing the first position). For example, the annotation information stored by the processor may include parameters representing a mapping between the second position and the second image. For example, the annotation information stored by the processor may include parameters representing a relationship between one or more third positions and one or more third images.

[0074] As described above, according to one embodiment, the processor of the electronic device may perform different types of calculations on multiple images included in the video to obtain annotation information. The annotation information obtained by the processor may be used for model training based on the multiple images. According to one embodiment, the processor of the electronic device may display a screen to receive user feedback on the annotation information, thereby improving the accuracy of the annotation information. For example, the processor may display a screen on a display (e.g., Figure 2 The processor may display the first image, any one of the one or more third images, and the second image on the display 230 in operation 340. The processor may display the first image, any one of the one or more third images, and the second image on the display in response to a second input indicating the playback of at least a portion of the video contained in the time period in operation 340. The processor may superimpose a visual object corresponding to any one of the first position, the third position, or the second position contained in the annotation information on the image displayed on the display. The visual object may have the form of a bounding box. Based on the visual object, the processor of the electronic device may receive an input for at least partially adjusting the annotation information. The input for at least partially adjusting the annotation information may be similar to the input for at least partially adjusting the annotation information. Figure 1b The illustrated representation selects an area (eg, areas 160 - 1 and 160 - 2 ) for input to be received.

[0075] Below, refer to Figure 4 , Figure 5a to Figure 5b , illustrating a method performed by an electronic device according to an embodiment Figure 3 Examples of operations 330 and 340 in FIG.

[0076] Figure 4 An example of an operation of an electronic device recognizing an object from an image included in a video according to an embodiment is shown. Figure 4 The operation of the electronic device described can be performed by Figure 1a to Figure 1b and Figure 2 The electronic device 101 and / or the processor 210 in the embodiment of the present invention are executed. Figure 4 The operation of the electronic device described can be Figure 3 At least one operation (eg, operation 330) is related.

[0077] Reference Figure 4 According to an embodiment, the electronic device may identify an external object (eg, a vehicle) that is commonly related to the first image 120-1 and the Nth image 120-N that are spaced apart from each other in a plurality of image sequences included in the video 120. The electronic device may Figure 3In operation 320, the first image 120-1 and the visual object 130-1 corresponding to the external object in the first image 120-1 are identified. In one embodiment, the electronic device may identify the position of the visual object 130-1 related to the external object in the first image 120-1 based on the input indicating the selection area 140-1. The input indicating the selection area 140-1 may include Figure 1b The position of the visual object 130-1 may correspond to Figure 3 first position.

[0078] Reference Figure 4 According to one embodiment, the electronic device may identify feature points (e.g., F11, F12, F13, F14, F15) contained in the region 140-1 based on identifying the region 140-1 of the first image 120-1. The electronic device may identify feature points (e.g., F11, F12, F13, F14, F15) based on color and / or brightness changes between pixels contained in the first image 120-1. For example, the feature points extracted from the region 140-1 may represent pixels with relatively drastic changes in color and / or brightness. For example, the feature points may include edges, corners, and / or skeletons represented by pixels contained in the region 140-1.

[0079] In addition, according to one embodiment, the electronic device can estimate the trajectory of pixels (Pixel) existing in the image frames constituting the video 120 by using a pixel trajectory estimation (Pixel Trajectory Estimation) method to track the motion trajectory of the external object contained in the area 140-1, thereby determining whether the external object existing in the area 140-1 of the first image 120-1 exists in the image (acquired) after the first image 120-1 (after t time), and predict the position of the object when it exists.

[0080] According to an embodiment, the electronic device may extract feature points from the Nth image 120-N at a second time point after the first time point of the first image 120-1 in the image sequence of the video 120. The electronic device may compare the feature points F11, F12, F13, F14, F15 extracted from the region 140-1 of the first image 120-1 with the feature points extracted from the Nth image 120-N, thereby identifying the region 140-N having a similar color and / or brightness to the region 140-1 in the Nth image 120-N. For example, the electronic device may identify feature points F21, F22, F23, F24, F25, F26 similar to the feature points F11, F12, F13, F14, F15 in the region 140-1 from the Nth image 120-N. Based on the identification of feature points F21, F22, F23, F24, F25, F26 from the Nth image 120-N, the electronic device can determine the area 140-N containing the feature points F21, F22, F23, F24, F25, F26 and / or the visual object 130-N contained in the area 140-N as a visual object related to the external object identified by the area 140-1 of the first image 120-1.

[0081] Although an embodiment has been described in which the electronic device identifies the second position related to the external object in the Nth image 120-N by comparing the feature points F11, F12, F13, F14, and F15 included in the area 140-1 of the first image 120-1 including the first position corresponding to the external object and one or more feature points included in the Nth image 120-N, the present embodiment is not limited thereto. According to an embodiment, the electronic device may input the first image 120-1 and the Nth image 120-N into a model for identifying an external object to identify the first position and the second position.

[0082] As described above, according to an embodiment of the electronic device, the image 120 may be extracted from the plurality of images in the video 120 in accordance with the user input (eg, reference Figure 1bThe electronic device may compare the feature points in the extracted image with the feature points F11, F12, F13, F14, and F15 within the area 140-1 of the first image 120-1, thereby identifying a position related to an external object (for example, an external object related to the area 140-1) in each of the extracted images. The position identified by the electronic device may be stored as annotation information corresponding to the video 120. According to an embodiment, the electronic device may repeatedly perform the first type of calculation based on feature points for identifying external objects in each of the time periods starting from the first time point of the first image 120-1. By repeatedly performing the first type of calculation, the electronic device may reduce the user input required to obtain the annotation information (for example, referring to Figure 1b Based on the linear interpolation of the positions identified by the calculation of the first type, the electronic device can obtain the annotation information of all images included in the video 120. Figure 5a to Figure 5b An operation example of an electronic device for acquiring annotation information based on the linear interpolation is described.

[0083] Figure 5a to Figure 5b An example of an electronic device operation for acquiring annotation information of other images based on annotation information of two images included in a video is shown. Figure 5a to Figure 5b The operation of the electronic device described can be performed by Figure 1a to Figure 1b and Figure 2 The electronic device 101 and / or the processor 210 executes. Figure 5a to Figure 5b The operation of the electronic device described can be Figure 3 At least one operation (eg, operation 340) is related.

[0084] Reference Figure 5a , showing that by executing Figure 4 The electronic device shown in the operation is identified and corresponds to the regions 140-1 and 140-N of the first image 120-1 and the Nth image 120-N, respectively. According to one embodiment, the regions 140-1 and 140-N corresponding to the external objects related to both the first image 120-1 and the Nth image 120-N can be identified by comparing one or more feature points included in the first image 120-1 with one or more feature points included in the Nth image 120-N.

[0085] Reference Figure 5aFor convenience of explanation, the positions of the regions 140-1 and 140-N respectively identified in the first image 120-1 and the Nth image 120-N are superimposed and displayed in the coordinate plane 510. The vertices of the coordinate plane 510 may be matched with the vertices of the first image 120-1 and the Nth image 120-N, respectively. According to one embodiment, the electronic device may obtain the annotation information of the image (e.g., the kth image 120-k) between the first image 120-1 and the Nth image 120-N based on the positions of the regions 140-1 and 140-N. For example, the annotation information obtained by the electronic device may correspond to the kth image 120-k between the first image 120-1 and the Nth image 120-N, and include a position related to an external object.

[0086] Reference Figure 5a , shows the region 520 in the k-th image 120-k obtained by interpolating the positions of the regions 140-1 and 140-N. According to an embodiment, the electronic device may obtain the coordinates of the vertices AK, BK, CK, and DK of the region 520 based on Mathematical Formula 1.

[0087]

Mathematical formula 1

[0088]

[0089] Referring to Formula 1, the x coordinate x of the vertex AK of the region 520 is AK The x coordinates x of the vertices A1 and AN of the region 140-1 of the first image 120-1 and the region 140-N of the Nth image 120-N may be A1 and x AN The value of the N:k segmentation. Similarly, the y coordinate y of the vertex AK of the region 520 is AK The y coordinates y of the vertices A1 and AN of the region 140-1 of the first image 120-1 and the region 140-N of the Nth image 120-N may be A1 and AN For example, the vertices AK, BK, CK, and DK of the region 520 may correspond to the inner points of the vertices A1, B1, C1, and D1 of the region 140-1 and the vertices AN, BN, CN, and DN of the region 140-N, respectively. Figure 5a , the vertices AK, BK, CK, and DK of the region 520 may be located on the lines connecting the vertices A1, B1, C1, and D1 of the region 140-1 to the vertices AN, BN, CN, and DN of the region 140-N, respectively. N in Mathematical Formula 1 represents the length of the time period between the first image 120-1 and the Nth image 120-N, and k in Mathematical Formula 1 represents the position of the time point corresponding to the kth image 120-k within the time period.

[0090] Reference Figure 5b , exemplarily illustrating the operation of the electronic device 101 to identify the region 520 from the kth image 120-k. The electronic device 101 can obtain the coordinates of the vertices AK, BK, CK, and DK of the region 520 of the kth image 120-k between the first image 120-1 to the Nth image 120-N based on Mathematical Formula 1. Figure 5b The x coordinate of vertex AK of region 520 may be a value obtained by dividing the x coordinate of vertex A1 of the region identified in first image 120-1 and the x coordinate of vertex AN of N-th image 120-N according to N:k. Similarly, the y coordinate of vertex AK of region 520 may be a value obtained by dividing the y coordinate of vertex A1 of the region identified in first image 120-1 and the y coordinate of vertex AN of N-th image 120-N according to N:k. The x coordinates of other vertices BK, CK, DK of region 520 may also be values ​​obtained by dividing the x coordinates of vertices B1, C1, D1 of the region identified in first image 120-1 and the x coordinates of vertices BN, CN, DN of the region identified in N-th image 120-N according to N:k. Similarly, the y coordinates of other vertices BK, CK, DK of area 520 may also be the values ​​obtained by dividing the y coordinates of vertices B1, C1, D1 of the area identified in the first image 120-1 and the y coordinates of vertices BN, CN, DN of the area identified in the Nth image 120-N according to N:k.

[0091] According to one embodiment, the electronic device may store data representing the position and / or size of the regions 140-1, 520, and 140-N as annotation information corresponding to the video 120. For example, the electronic device may store data of vertices A1, B1, C1, and D1 in the annotation information as information representing the region 140-1 corresponding to the first image 120-1. The electronic device may store coordinate data of vertices AK, BK, CK, and DK in the annotation information as information representing the region 140-k corresponding to the kth image 120-k. The electronic device may store parameters related to at least one of vertices AN, BN, CN, and DN in the annotation information as information representing the region 140-N corresponding to the Nth image 120-N. The electronic device may further store parameters indicating that the parameters correspond to the Nth image 120-N in the annotation information together with parameters related to at least one of vertices AN, BN, CN, and DN of the region 140-N.

[0092] As described above, according to an embodiment, the electronic device may obtain the third coordinate of the third position of the area 520 in the k-th image 120-k between the first image 120-1 and the N-th image 120-N by interpolating the first coordinate of the first position of the area 140-1 in the first image 120-1 at the first time point and the second coordinate of the second position of the area 140-N in the N-th image 120-N at the second time point using the length between the first time point and the second time point (e.g., N in Mathematical Formula 1). The electronic device may obtain the third position by interpolating the first coordinate and the second coordinate based on the time point of the k-th image 120-k (e.g., k in Mathematical Formula 1). For example, the electronic device may perform a second type of calculation (e.g., the interpolation described with reference to Mathematical Formula 1) based on the first position, the second position, and the time point of the k-th image to obtain the third position.

[0093] Below, we will refer to Figure 6 Exemplary operations of the electronic device described with reference to the aforementioned drawings are described.

[0094] Figure 6 An example of an operation flow chart performed by an electronic device according to an embodiment is shown. Figure 6 The operation of the electronic device can be performed by Figure 1a to Figure 1b and Figure 2 The electronic device 101 and / or the processor 210 executes. Figure 6 The operation of the electronic device can be compared with Figure 3 The operation of the electronic device described is related.

[0095] Reference Figure 6 In operation 610, a processor of an electronic device according to an embodiment may identify a plurality of images for playing a video. The processor may be similar to Figure 3 Operation 310 performs Figure 6 Operation 610. The video in operation 610 may include Figures 1a to 5a Video 120 in.

[0096] Reference Figure 6 In operation 620, the processor of the electronic device according to an embodiment may obtain the annotation data corresponding to the ath image among the multiple images. The ath image in operation 620 may include Figures 1a to 5b The annotation data in operation 620 may represent an area in the a-th image that is related to an external object (eg, Figures 1a to 5a The processor can Figure 1a The screen 110 is used to obtain the annotation data based on the input indicating the selection of the area in the ath image. The input may include reference Figure 1bThe processor may be based on the location of the area (e.g., Figures 1a to 5a The annotation data is obtained by inputting the coordinates of the vertices A1, B1, C1, and D1 of the area 140-1.

[0097] Reference Figure 6 In operation 630, the processor of the electronic device according to an embodiment may obtain the annotation data corresponding to the a+Mth image among the multiple images. The a+Mth image in operation 630 may include Figures 1a to 5b The annotation data acquired by the processor in operation 630 may represent an area in the a+Mth image related to an external object (e.g., Figures 1a to 5a According to one embodiment, the processor may be based on the reference Figure 4 Describe the operation, perform Figure 6 Operation 630 in .

[0098] For example, in Figure 6 In operation 632 of the present invention, the processor of the electronic device according to an embodiment may compare feature points of the a-th image and feature points of the a+M-th image to identify the position of an external object related to the a-th image in the a+M-th image. Based on the similarity between the a-th image and the a+M-th image represented by at least one feature point, the processor may identify the positions of the external object contained in the a-th image and the a+M-th image in the a-th image and the a+M-th image, respectively. In an embodiment, if the position in the a+M-th image related to the external object cannot be automatically identified based on the feature points, the processor may request the user of the electronic device to input the position.

[0099] For example, in Figure 6 In operation 634, according to an embodiment, the processor of the electronic device may identify whether the visual object corresponding to the external object is included in the a+Mth image. For example, by comparing one or more feature points identified from a position in the a+Mth image related to the external object represented by the annotation data of operation 620 with one or more feature points included in the a+Mth image, the processor may identify whether the visual object corresponding to the external object is included in the a+Mth image. In an embodiment using a model for identifying an external object, the processor may identify whether the visual object corresponding to the external object is included in the a+Mth image based on the probability that the a+Mth image includes the external object identified by the model. If the visual object is included in the a+Mth image (634-Yes), the processor may perform operation 640. If the visual object is not included in the a+Mth image (634-No), the processor may perform operation 636.

[0100] For example, in Figure 6 In operation 636, a processor of an electronic device according to an embodiment may obtain annotation data corresponding to the (a+M)th image based on an input representing a position selected in the (a+M)th image related to an external object. In operation 636, the processor may display, on a display, a screen for obtaining a position related to the external object in the (a+M)th image. Based on the input received through the screen, the processor may obtain the annotation data of operation 636.

[0101] Referring to Figure 6 , in operation 640, a processor of an electronic device according to an embodiment may generate annotation data corresponding to the (a+N)th image based on the annotation data corresponding to the ath image and the annotation data corresponding to the (a+M)th image. N may be a natural number greater than 0 and less than M (0 < N < M). The (a+N)th image of operation 640 may include Figures 1a to 5b the kth image in Figure 5a and / or Figure 5b described operations, the processor may perform Figure 6 operation 640. The processor may generate annotation data corresponding to the (a+N)th image by performing interpolation on the position in the ath image represented by the annotation data of operation 620 and the position in the (a+M)th image represented by the annotation data of operation 630. The processor may perform operation 640 based on the operations described with reference to

[0102] According to an embodiment, a processor of an electronic device may obtain all annotation data from the ath image to the (a+M)th image by performing Figure 6 operations 610 to 640. Referring to Figure 6 , in operation 650, a processor of an electronic device according to an embodiment may calibrate the annotation data generated through operations 610 to 640. The calibration performed by the processor may be for combining the annotation data linearly adjusted through linear interpolation with the non-linear motion of an external object captured by a video.

[0103] For example, in Figure 6In operation 652, according to an embodiment, the processor of the electronic device may identify whether a parameter (e.g., a natural number M) used to represent the interval between the a-th image and the a+M-th image exceeds a threshold value. The interval may be associated with the length of the time period between the time point of the a-th image and the time point of the a+M-th image. If the interval exceeds the threshold value (652-yes), the processor may perform operation 654. For example, when the interval is longer than the threshold interval, the processor may perform operation 654. If the interval does not exceed the threshold value (652-no), the processor may perform operation 656.

[0104] For example, in Figure 6 In operation 654, the processor of the electronic device according to one embodiment may change the annotation data corresponding to the a+Nth image based on the feature points. For example, the processor may compare one or more feature points extracted from the a+Nth image with feature points included in the a+Nth image based on the annotation data corresponding to the a+Nth image (for example, annotation data obtained from a user of the electronic device), thereby changing the annotation data corresponding to the a+Nth image. The one or more feature points extracted from the a+Nth image may be extracted from an area related to an external object in the a+Nth image represented by the annotation data. The feature points in the a+Nth image compared with the one or more feature points extracted from the a+Nth image may be extracted from the entirety of the a+Nth image.

[0105] For example, the processor may identify, based on the comparison, an area in the a+Nth image that is similar to an area of ​​the a+Nth image represented by the annotation data corresponding to the a+Nth image. The processor may change the annotation data corresponding to the a+Nth image based on the position of the area in the a+Nth image identified based on the comparison. The processor may change the annotation data generated by the operation 640 related to interpolation based on the operation 654 related to comparing feature points of different images.

[0106] For example, in Figure 6 In operation 656, the processor of the electronic device according to an embodiment may change the annotation data based on the input received through the screen for displaying the video and the annotation data. The screen of operation 656 may include Figure 1a The processor of the electronic device may identify, through the screen, an input for changing the annotation data corresponding to any one of the a-th image to the a+M-th image. Based on the input, the processor may adjust the annotation data corresponding to the input, thereby changing the position in the image related to the external object.

[0107] Reference Figure 6In operation 660, the processor of the electronic device according to an embodiment may store the annotation data of the ath image, the a+Nth image, and the a+Mth image in association with the video. The annotation data stored in association with the video in operation 660 may correspond to Figure 2 The annotation information 240. The annotation data stored by operation 660 can be used together with the video associated with the annotation data in the training of a model (or neural network) for recognizing videos and / or images.

[0108] In one embodiment, the annotated data identified based on operation 660 may include information as shown in Table 1.

[0109]

Table 1

[0110]

[0111] The information of the external object (Shapes) in Table 1 may include information of the external object contained in the image as a subordinate object. The information of the external object in Table 1 may include information as shown in Table 2.

[0112]

Table 2

[0113]

[0114] In one embodiment, K-fold cross validation may be performed for training of the neural network. Figure 6 The processor that has obtained the annotation data corresponding to a large number of images can execute an algorithm such as K-fold cross validation to verify the credibility of the neural network. For example, the processor can divide the annotation data into a training set, a validation set and / or a test set. For example, the processor can distinguish the annotation data based on the image flags in Table 1. For example, according to the image flags based on weather, the processor can classify the annotation data according to weather. The operation of classifying the annotation data based on image flags can be used to uniformize the characteristics of the image used for training and / or verifying the neural network. For example, using the annotation data classified according to the image flags based on weather, the processor can sample the annotation data from the annotation data group corresponding to different weathers so that the annotation data classified as a specific weather is not used for the training and / or verification of the neural network. If the annotation data is generated for more than a specified number of images, the processor can divide the annotation data into three sets such as a training set, a validation set and a test set. If the annotation data is generated for the image below the specified number, the processor can divide the annotation data into two sets such as a training set and a test set.

[0115] In one embodiment, the processor may divide the labeled data contained in the training set into K folds. Within the divided folds, the processor may divide the labeled data into K folds again, and then designate K-1 labeled data as labeled data for training, and designate the remaining 1 labeled data as labeled data for verification. The processor may generate a neural network (or model) and input the labeled data for the training to extract error values. The processor may cross the labeled data for verification according to each fold to extract error values. After extracting the error values ​​of all folds, the processor may optimize the neural network based on the extracted error values. The processor may train the entire training set based on the optimized neural network. After training the entire training set, the processor may evaluate the neural network based on the labeled data contained in the test set.

[0116] As described above, according to an embodiment of the present invention, the processor of the electronic device can alternately perform a first type of calculation based on feature point comparison and a second type of calculation based on interpolation between positions (or coordinates) represented by the annotation data to obtain annotation data of all multiple images included in the video. Since the processor alternately performs the calculation, it is not necessary to extract feature points from all multiple images, so that the processor can obtain the annotation data more quickly.

[0117] Below, we will refer to Figure 7 An example of the operation of an electronic device for recognizing a user input for changing annotation data is described.

[0118] Figure 7 FIG. 2 shows an example of interpolation performed by an electronic device according to an embodiment. Figure 7 The operation of the electronic device described can be performed by Figure 1a to Figure 1b , Figure 2 The electronic device 101 and / or the processor 210 executes. Figure 7 The operation of the electronic device described can be Figure 3 and / or Figure 6 At least one of the operations (for example, Figure 6 is related to at least one of operations 654, 656).

[0119] Reference Figure 7 , showing regions 140-1, 520, 140-N corresponding to the first image 120-1, the kth image 120-k, and the Nth image 120-N respectively included in the sequence of multiple images for playing the video 120. The region 140-1 corresponding to the first image 120-1 can be selected by a user input (e.g., referring to Figure 1bThe region 140-N corresponding to the Nth image 120-N can be identified based on a comparison of one or more feature points included in the region 140-1 of the first image 120-1 and feature points included in the Nth image 120-N. The region 520 corresponding to the kth image 120-k can be identified based on a comparison of one or more feature points included in the region 140-1 of the first image 120-1 and feature points included in the Nth image 120-N. Figure 5a to Figure 5b For example, the electronic device may identify the position of the region 520 of the k-th image 120 - k based on the interpolation of the region 140 - 1 and the region 140 -N.

[0120] exist Figure 7 In the exemplary case of , based on the nonlinear motion of the external object shown in the video 120, the position of the region 520 obtained by linear interpolation may deviate from the position related to the external object in the k-th image 120-k. According to one embodiment, the processor of the electronic device may be based on Figure 6 In operation 654, the feature points included in the k-th image 120-k are compared with the feature points of other images (e.g., the first image 120-1) (e.g., the feature points in the region 140-1 of the first image 120-1), thereby changing the region 520 identified by the interpolation. Figure 7 , the processor may change the region related to the external object in the kth image 120-k from region 520 to region 720 based on the comparison of the feature points. The processor may perform Figure 6 In operation 654 , the region 520 obtained based on the interpolation is changed and / or moved to the region 720 .

[0121] exist Figure 7 In the exemplary case of , the processor may recognize input indicating a change to the region 520 of the kth image 120-k. For example, in the example of Figure 2 In a state where the display 230 of the processor 230 displays the kth image 120-k, the processor can, based on the annotation information corresponding to the kth image 120-k, superimpose on the kth image 120-k a visual object (e.g., a bounding box) for indicating the position of the external object. The position of the visual object on the kth image 120-k can correspond to the area 520. In the example, the processor can recognize an input indicating the movement of the visual object. Based on the input, the processor can adjust the annotation data corresponding to the kth image 120-k according to the position of the visual object moved by the input. For example, based on the input of moving the area 520 to the area 720, the processor can change the position related to the external object in the kth image 120-k to the position of the area 720 in the annotation data.

[0122] According to one embodiment, the processor of the electronic device may change the annotation data of other images adjacent to the kth image 120-k in the plurality of image sequences for playing the video 120 based on changing the position related to the external object in the kth image 120-k and using the changed position. For example, in a state where the position related to the external object in the kth image 120-k is changed from a position related to the area 520 to a position related to the area 720, the processor may change the position related to the external object in at least one image different from the kth image 120-k. For example, the processor may change the annotation data corresponding to other images between the first image 120-1 and the kth image 120-k (e.g., the k-1th image 120-k-1) and / or other images between the kth image 120-k and the Nth image 120-N (e.g., the k+1th image 120-k+1).

[0123] Reference Figure 7 , in a state where the region 520 of the k-th image 120-k represented by the annotation data obtained based on the interpolation is changed to the region 720, the processor can identify the position and / or region related to the external object in other images between the first image 120-1 and the k-th image 120-k on the line connecting the vertices AK2, BK2, CK2, DK2 of the region 720 and the vertices A1, B1, C1, D1 of the first image 120-1. For example, the processor can obtain the annotation data of the images between the first image 120-1 and the k-th image 120-k based on the interpolation of the region 140-1 of the first image 120-1 and the region 720 of the k-th image 120-k. Similarly, in the described state, the processor can identify positions and / or areas related to external objects in other images between the kth image 120-k and the Nth image 120-N on the line connecting the vertices AK2, BK2, CK2, DK2 of the area 720 and the vertices AN, BN, CN, DN of the Nth image 120-N.

[0124] As described above, based on the input of changing the region 520 of the k-th image 120-k represented by the annotation data obtained by interpolation to the region 720, the electronic device 101 can further adjust the annotation data of other images adjacent to the k-th image 120-k (for example, images between the first image 120-1 and the k-th image 120-k and / or other images between the k-th image 120-k and the N-th image 120-N). Based on the above adjustment, the electronic device 101 can use the input to improve the accuracy of the annotation information for the multiple images included in the video 120.

[0125] As described above, according to an embodiment of the electronic device, in order to more quickly obtain the annotation information of multiple images included in the video 120, feature point matching and / or linear interpolation can be selectively applied to the multiple images. The electronic device can reduce the amount of calculation and / or time for generating the annotation information based on the interpolation, and can maintain and / or improve the accuracy of the annotation information based on the feature point matching.

[0126] Figure 8 An example block diagram of an autonomous driving system for a vehicle according to an embodiment is shown.

[0127] Fig. 9 and Fig.10 An example block diagram of an autonomous driving mobile body according to an embodiment is shown.

[0128] Fig.11 An example of a gateway associated with a user equipment according to an embodiment is shown.

[0129] like Figure 8 As shown, the vehicle's autonomous driving system 800 may be a deep learning network including a sensor 803, an image preprocessor 805, a deep learning network 807, an artificial intelligence (AI) processor 809, a vehicle control module 811, a network interface 813, and a communication unit 815. In various embodiments, the elements may be connected through various interfaces. For example, the sensor data sensed and output by the sensor 803 may be transmitted (feed) to the image preprocessor 805. The sensor data processed by the image preprocessor 805 may be transmitted (feed) to the deep learning network 807 run by the AI ​​processor 809. The output of the deep learning network 807 run by the AI ​​processor 809 may be transmitted (feed) to the vehicle control module 811. The intermediate results of the deep learning network 807 run on the AI ​​processor 809 may be transmitted (feed) to the AI ​​processor 809. In various embodiments, the network interface 813 communicates with the in-vehicle electronic device, thereby delivering the autonomous driving path information and / or autonomous driving control commands for the vehicle's autonomous driving to the internal module. In one embodiment, the network interface 813 can be used to transmit sensor data acquired by the sensor 803 to an external server. In some embodiments, the autonomous driving control system 800 can include additional or fewer elements as appropriate. For example, in some embodiments, the image preprocessor 805 can be an optional element. According to another example, a post-processing element (not shown) can be included in the autonomous driving control system 800 to post-process the output of the deep learning network 807 before the output is provided to the vehicle control module 811.

[0130] In some embodiments, the sensor 803 may include more than one sensor. In various embodiments, the sensor 803 may be attached to different locations of the vehicle. The sensor 803 may face more than one different direction. For example, the sensor 803 may be attached to the front, sides, rear, and / or roof of the vehicle to face the forward-facing, rear-facing, side-facing, and other directions. In some embodiments, the sensor 803 may be an image sensor such as a high dynamic range camera. In some embodiments, the sensor 803 includes non-visual sensors. In some embodiments, the sensor 803 includes a radar, a laser radar (LiDAR, Light Detection And Ranging), and / or an ultrasonic sensor in addition to an image sensor. In some embodiments, the sensor 803 is not mounted on a vehicle having the vehicle control module 811. For example, the sensor 803 is used as part of a deep learning system to capture sensor data, and may be attached to the environment or road, and / or mounted on surrounding vehicles.

[0131] In some embodiments, an image preprocessor 805 may be used to preprocess sensor data from the sensor 803. For example, the image preprocessor 805 may be used to preprocess sensor data, split the sensor data into one or more constituent elements, and / or post-process one or more constituent elements. In some embodiments, the image preprocessor 805 may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image preprocessor 805 may be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image preprocessor 805 may be a constituent element of the AI ​​processor 809.

[0132] In some embodiments, the deep learning network 807 may be a deep learning network for implementing control commands for controlling an autonomous vehicle. For example, the deep learning network 807 may be an artificial neural network such as a convolutional neural network (CNN) trained using sensor data, and the output of the deep learning network 807 may be provided to the vehicle control module 811.

[0133] In some embodiments, the artificial intelligence (AI) processor 809 can be a hardware processor for running the deep learning network 807. In some embodiments, the AI ​​processor 809 is a specialized AI processor for performing inference of a convolutional neural network (CNN) on sensor data. In some embodiments, the AI ​​processor 809 can be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 809 can be optimized for deep learning operations, such as neural network operations including convolution, inner product, vector and / or matrix operations. In some embodiments, the AI ​​processor 809 can be implemented by multiple graphics processing units (GPUs) that can effectively perform parallel processing.

[0134] In various embodiments, the AI ​​processor 809 performs deep learning analysis on the sensor data received from the sensor 803 during the execution of the AI ​​processor 809, and can be coupled to a memory through an input / output interface, the memory being configured to provide an AI processor with instructions, through which the machine learning results for making the vehicle at least partially autonomous can be determined. In some embodiments, the vehicle control module 811 can process the instructions for vehicle control output by the artificial intelligence (AI) processor 809, and convert the output of the AI ​​processor 809 into instructions for controlling various modules of the vehicle to control various modules of the vehicle. In some embodiments, the vehicle control module 811 is used to control an autonomous driving vehicle. In some embodiments, the vehicle control module 811 can adjust the steering and / or speed of the vehicle. For example, the vehicle control module 811 can be used to control the vehicle's driving operations such as deceleration, acceleration, steering, lane change, and lane keeping. In some embodiments, the vehicle control module 811 can generate control signals for controlling vehicle lighting such as brake lights, turn signals, and headlights. In some embodiments, the vehicle control module 811 can be used to control vehicle audio-related systems such as the vehicle's sound system, the vehicle's audio warnings, the vehicle's microphone system, and the vehicle's horn system.

[0135] In some embodiments, the vehicle control module 811 can be used to control notification systems, which include warning systems designed to inform passengers and / or drivers of driving events such as approaching a predetermined destination or the presence of a potential collision. In some embodiments, the vehicle control module 811 can be used to adjust sensors such as the vehicle's sensor 803. For example, the vehicle control module 811 can modify the orientation of the sensor 803, change the output resolution and / or format type of the sensor 803, increase or decrease the capture rate, adjust the dynamic range, and adjust the focus of the camera. In addition, the vehicle control module 811 can turn on / off the operation of sensors individually or collectively.

[0136] In some embodiments, the vehicle control module 811 may change the parameters of the image preprocessor 805 by modifying the frequency range of the filter, adjusting the edge detection parameter for detecting features and / or objects, adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 811 may be used to control the autonomous driving and / or driver assistance functions of the vehicle.

[0137] In some embodiments, the network interface 813 may be responsible for the internal interface between the modules of the autonomous driving control system 800 and the communication unit 815. Specifically, the network interface 813 may be a communication interface for receiving and / or sending data including voice data. In various embodiments, the network interface 813 may connect voice calls, receive and / or send text messages, transmit sensor data, update the vehicle's autonomous driving system software, or connect to an external server for updating the vehicle's autonomous driving system software through the communication unit 815.

[0138] In various embodiments, the communication unit 815 may include various wireless interfaces in the form of cellular or WiFi. For example, the network interface 813 may receive updates of operating parameters and / or instructions for the sensor 803, the image preprocessor 805, the deep learning network 807, the AI ​​processor 809, and the vehicle control module 811 from an external server connected via the communication unit 815. For example, the machine learning model of the deep learning network 807 may be updated using the communication unit 815. According to another example, the communication unit 815 may be used to update operating parameters of the image preprocessor 805 such as image processing parameters and / or firmware of the sensor 803.

[0139] In other embodiments, the communication unit 815 can be used to activate communication with emergency services to obtain emergency contact in the event of an accident or near-accident. For example, in the event of a collision, the communication unit 815 can be used to call emergency services for help and inform the emergency service of the details of the collision and the location of the vehicle. In various embodiments, the communication unit 815 can update or obtain an estimated time of arrival and / or destination location.

[0140] According to one embodiment, Figure 8The autonomous driving system 800 shown may be configured as an electronic device of a vehicle. According to one embodiment, if an autonomous driving release event from a user occurs during the autonomous driving of the vehicle, the AI ​​processor 809 of the autonomous driving system 800 may control the input of information related to the autonomous driving release event into the training set data of the deep learning network to enable the autonomous driving software of the vehicle to learn.

[0141] Fig. 9 and Fig.10 An example of a block diagram for representing an autonomous driving mobile body according to an embodiment is shown. Fig. 9 According to this embodiment, the autonomous driving mobile body 900 may include a control device 1000, sensor modules 904a, 904b, 904c, 904d, an engine 906 and a user interface 908.

[0142] The autonomous driving mobile body 900 may have an autonomous driving mode or a manual mode. For example, the autonomous driving mode may be switched from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode according to user input received through the user interface 908 .

[0143] When the mobile body 900 operates in the automatic driving mode, the automatic driving mobile body 900 may operate under the control of the control device 1000 .

[0144] The control device 1000 in this embodiment may include a controller 1020 having a memory 1022 and a processor 1024 , a sensor 1010 , a communication device 1030 , and an object detection device 1040 .

[0145] Here, the object detection device 1040 may perform all or part of the functions of the distance measurement device (eg, the electronic device 101 ).

[0146] That is, in this embodiment, the object detection device 1040 is a device for detecting an object located outside the moving body 900. The object detection device 1040 can detect the object located outside the moving body 900 and generate object information according to the detection result.

[0147] The object information may include information on the presence or absence of the object, position information of the object, distance information between the moving body and the object, and relative speed information between the moving body and the object.

[0148] The object may include various objects outside the moving body 900, such as lanes, other vehicles, pedestrians, traffic signals, light, roads, structures, speed bumps, terrain objects, animals, etc. Here, the traffic signal may be a traffic light, a traffic sign, a pattern or text on the road surface, etc. Also, the light may be light generated by lights on other vehicles, light generated by street lamps, or sunlight.

[0149] Furthermore, structures may be objects located around the road and fixed to the ground. For example, structures may include street lamps, roadside trees, buildings, electric poles, signal lights, bridges, etc. Terrain objects may include mountains, hills, etc.

[0150] The object detection device 1040 may include a camera module. The controller 1020 may extract object information from an external image captured by the camera module and process the information.

[0151] In addition, the object detection device 1040 may also include an imaging device for sensing the external environment. In addition to LIDAR, RADAR, GPS devices, Odometry and other computer vision devices, ultrasonic sensors, infrared sensors, etc. may also be used, and these devices may be operated selectively or simultaneously as needed to achieve more accurate sensing.

[0152] In addition, the distance measuring device according to an embodiment of the present invention can calculate the distance between the autonomous driving mobile body 900 and the object, and is connected to the control device 1000 of the autonomous driving mobile body 900 to control the movement of the mobile body according to the calculated distance.

[0153] As an example, when the distance between the autonomous driving mobile body 900 and the object is such that there is a possibility of collision, the autonomous driving mobile body 900 may decelerate or stop by controlling the brakes. As another example, when the object is a moving object, the autonomous driving mobile body 900 may control the driving speed of the autonomous driving mobile body 900 to maintain a predetermined distance from the object.

[0154] The distance measuring device according to an embodiment of the present invention may be configured as a module in the control device 1000 of the autonomous driving mobile body 900. That is, the collision avoidance method of the present invention may be implemented in software by the memory 1022 and the processor 1024 of the control device 1000.

[0155] In addition, the sensor 1010 is connected to the sensor modules 904a, 904b, 904c, and 904d, so that various sensor information of the internal / external environment of the moving body can be obtained. Here, the sensor 1010 may include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight detection sensor, a heading sensor, a gyro sensor, a position module, a moving body forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor for steering wheel rotation, a moving body internal temperature sensor, a moving body internal humidity sensor, an ultrasonic sensor, a light sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0156] Therefore, sensor 1010 can obtain sensor signals such as mobile body posture information, mobile body collision information, mobile body direction information, mobile body position information (GPS information), mobile body angle information, mobile body speed information, mobile body acceleration information, mobile body tilt information, mobile body forward / backward information, battery information, fuel information, tire information, mobile body light information, mobile body internal temperature information, mobile body internal humidity information, steering wheel rotation angle, mobile body external brightness, pressure applied on the accelerator pedal, pressure applied on the brake pedal, etc.

[0157] In addition, sensor 1010 may also include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crankshaft angle sensor (CAS), etc.

[0158] In this way, the sensor 1010 can generate moving body state information based on the sensing data.

[0159] The wireless communication device 1030 is configured to implement wireless communication between the autonomous driving mobile bodies 900. For example, the autonomous driving mobile body 900 communicates with the user's mobile phone or other wireless communication devices 1030, other mobile bodies, central devices (traffic control devices), servers, etc. The wireless communication device 1030 can send and receive wireless signals according to the access wireless protocol. The wireless communication protocol can be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (GSM), but the communication protocol is not limited to this.

[0160] In addition, in this embodiment, the autonomous driving mobile body 900 can also achieve communication between mobile bodies through the wireless communication device 1030. That is, the wireless communication device 1030 can communicate with other mobile bodies on the road and other mobile bodies through vehicle-to-vehicle (V2V) communication. The autonomous driving mobile body 900 can send and receive information such as driving warnings, traffic information, etc. through communication between vehicles, and can also request information from other mobile bodies or receive requests. For example, the wireless communication device 1030 can perform V2V communication through a short-range communication (DSRC, dedicated short-range communication) device or a C-V2V (Cellular-V2V) device. In addition, in addition to vehicle-to-vehicle communication, communication (V2X, Vehicle to Everything communication) between vehicles and other things (such as electronic devices carried by pedestrians, etc.) can also be achieved through the wireless communication device 1030.

[0161] In this embodiment, the controller 1020 is a unit that controls the overall operation of each unit in the mobile body 900, and can be configured by the manufacturer of the mobile body during manufacturing, or additionally configured after manufacturing to perform the automatic driving function. Alternatively, it can include a configuration that performs continuous additional functions by upgrading the controller 1020 configured during manufacturing. Such a controller 1020 can also be called an ECU (Electronic Control Unit).

[0162] The controller 1020 can collect various data from the connected sensors 1010, the object detection device 1040, the communication device 1030, etc., and transmit control signals based on the collected data to the sensors 1010, the engine 906, the user interface 908, the communication device 1030, and the object detection device 1040 included as elements of the mobile body. In addition, although not shown in the figure, the controller 1020 can also transmit control signals to an acceleration device, a braking system, a steering device, or a navigation device related to the travel of the mobile body.

[0163] In this embodiment, the controller 1020 can control the engine 906. For example, the autonomous driving mobile body 900 senses the speed limit of the road on which it is traveling, and controls the engine 906 to ensure that the driving speed does not exceed the speed limit, or controls the engine 906 to accelerate the driving speed of the autonomous driving mobile body 900 within a range that does not exceed the speed limit.

[0164] In addition, if the autonomous driving mobile body 900 approaches a lane line or deviates from a lane line during driving, the controller 1020 determines whether such approaching and deviating from the lane line is caused by a normal driving condition or other driving conditions, and controls the engine 906 according to the determination result to control the driving of the mobile body. Specifically, the autonomous driving mobile body 900 can detect lane lines formed on both sides of the driving lane of the mobile body. In this case, the controller 1020 can determine whether the autonomous driving mobile body 900 approaches a lane line or deviates from a lane line. If it is determined that the autonomous driving mobile body 900 approaches a lane line or deviates from a lane line, it can further determine whether such driving is caused by a normal driving condition or other driving conditions. Here, the normal driving condition can be, for example, a condition in which the mobile body needs to change lanes. And other driving conditions can be, for example, a condition in which the mobile body does not need to change lanes. If the controller 1020 determines that the autonomous driving mobile body 900 approaches a lane line or deviates from a lane line when the mobile body does not need to change lanes, the driving of the autonomous driving mobile body 900 can be controlled so that the autonomous driving mobile body 900 does not deviate from the lane line and drives normally.

[0165] In the case where there are other moving objects or obstacles in front of the moving object, the moving object can be decelerated by controlling the engine 906 or the braking system. In addition to the speed, the trajectory, driving path, and steering angle can also be controlled. Alternatively, the controller 1020 can generate necessary control signals based on the sensing information of the moving object's driving lane line, driving signal, and other external environments to control the driving of the moving object.

[0166] In addition to generating its own control signals, the controller 1020 can also communicate with surrounding mobile objects or a central server, and send instructions for controlling surrounding devices through the received information to control the travel of the mobile object.

[0167] In addition, when the position or perspective of the camera module 1050 changes, it may be difficult to accurately identify the moving object or lane line. Therefore, in order to prevent this from happening, the controller 1020 may also generate a control signal to perform calibration of the camera module 1050. Therefore, in this embodiment, the controller 1020 sends a calibration control signal to the camera module 1050, so that even if the installation position of the camera module 1050 changes due to vibration or impact caused by the movement of the autonomous driving moving object 900, the normal installation position, direction and perspective of the camera module 1050 can be continuously maintained. The controller 1020 can generate a control signal to perform calibration of the camera module 1050 when the difference between the initial installation position, direction, and perspective information of the camera module 1050 stored in advance and the initial installation position, direction, and perspective information of the camera module 1050 measured during the driving of the autonomous driving moving object 900 exceeds a threshold.

[0168] The controller 1020 in this embodiment may include a memory 1022 and a processor 1024. The processor 1024 may execute the software stored in the memory 1022 according to the control signal of the controller 1020. Specifically, the controller 1020 may store data and instructions for executing the lane line detection method of the present invention in the memory 1022, and the processor 1024 may execute these instructions to implement one or more methods disclosed herein.

[0169] At this time, the memory 1022 may be stored in a non-volatile recording medium executable by the processor 1024. The memory 1022 may store software and data through appropriate internal and external devices. The memory 1022 may be composed of a RAM (random access memory), a ROM (read only memory), a hard disk, and a memory 1022 device connected to a dongle.

[0170] The memory 1022 can store at least an operating system (OS), user applications, and executable commands. The memory 1022 can also store application data, array data structures, and the like.

[0171] Processor 1024 is a microprocessor or suitable electronic processor and may be a controller, microcontroller or state machine.

[0172] The processor 1024 may be implemented by a combination of computing devices, which may be a digital signal processor, a microprocessor, or a suitable combination thereof.

[0173] On the other hand, the autonomous driving mobile body 900 may further include a user interface 908 for inputting user input to the control device 1000. The user interface 908 may allow the user to input information through an appropriate interactive method. For example, it may be implemented by a touch screen, a keyboard, an operation button, etc. The user interface 908 transmits the input or instruction to the controller 1020, and the controller 1020 performs the control operation of the mobile body in response to the input or instruction.

[0174] In addition, the user interface 908 can be a device external to the autonomous driving mobile body 900, and communicate with the autonomous driving mobile body 900 through the wireless communication device 1030. For example, the user interface 908 can be linked with a mobile phone, a tablet computer, or other computing devices.

[0175] In addition, the autonomous driving mobile body 900 described in this embodiment includes an engine 906, but may also include other types of propulsion systems. For example, the mobile body may be driven by electric energy, or by a hybrid power system of hydrogen energy or a combination thereof. Therefore, the controller 1020 may include a propulsion mechanism according to the propulsion system of the autonomous driving mobile body 900, and provide corresponding control signals to the components of each propulsion mechanism.

[0176] Below, refer to Fig.10 , the detailed structure of the control device 1000 according to this embodiment is described in more detail.

[0177] The control device 1000 includes a processor 1024. The processor 1024 may be a general-purpose single-chip or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor may also be referred to as a central processing unit (CPU). In this embodiment, the processor 1024 may also use a combination of multiple processors.

[0178] The control device 1000 further includes a memory 1022. The memory 1022 may be any electronic component capable of storing electronic information. The memory 1022 may not only be a single memory but may also include a combination of multiple memories 1022.

[0179] The data and instructions 1022a required for the distance measurement device of the present invention to execute the distance measurement method may be stored in the memory 1022. When the processor 1024 executes the instruction 1022a, all or part of the instruction 1022a and the data 1022b required to execute the instruction may be loaded (1024a, 1024b) onto the processor 1024.

[0180] The control device 1000 may include a transmitter 1030a, a receiver 1030b, or a transceiver 1030c to allow transmission and reception of signals. One or more antennas 1032a, 1032b may be electrically connected to the transmitter 1030a, the receiver 1030b, or each transceiver 1030c, and may further include an antenna.

[0181] The control device 1000 may further include a digital signal processor (DSP) 1070. Through the DSP 1070, the moving body can quickly process digital signals.

[0182] The control device 1000 may further include a communication interface 1080. The communication interface 1080 may include one or more ports and / or communication modules for connecting other devices to the control device 1000. The communication interface 1080 may enable a user to interact with the control device 1000.

[0183] The components of the control device 1000 may be connected together via one or more buses 1090 , and the bus 1090 may include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 1024 , the components may communicate information to each other via the bus 1090 to perform the intended functions.

[0184] On the other hand, in various embodiments, the control device 1000 can be associated with a gateway to communicate with a secure cloud. Fig.11 , the control device 1000 may be associated with a gateway 1105 so as to provide the security cloud 1106 with information acquired from at least one of the components 1101 to 1104 of the vehicle 1100. For example, the gateway 1105 may be included in the control device 1000. According to another example, the gateway 1105 may be configured as an independent device in the vehicle 1100 that is different from the control device 1000. The gateway 1105 connects the software management cloud 1109 having different networks, the security cloud 1106, and the network in the vehicle 1100 that is secured by the in-vehicle security software 1110, so that they can communicate.

[0185] For example, the component 1101 may be a sensor. For example, the sensor may be used to obtain information related to at least one of the state of the vehicle 1100 or the state around the vehicle 1100. For example, the component 1101 may include the sensor 1410.

[0186] For example, component 1102 may be an electronic control unit (ECU). For example, the ECU may be used for engine control, transmission control, airbag control, and tire pressure management.

[0187] For example, component 1103 may be an instrument cluster. For example, the instrument cluster may refer to a panel in a dashboard located directly in front of the driver. For example, the instrument cluster may be configured to display information required for driving to the driver (or passenger). For example, the instrument cluster may be used to display a visual element indicating the engine revolutions per minute (RPM), a visual element indicating the speed of the vehicle 1100, a visual element indicating the amount of remaining fuel, a visual element indicating the gear state, or at least one of the visual elements used to indicate information obtained through component 1101.

[0188] For example, component 1104 may be a telematics device. For example, the telematics device may refer to a device that combines wireless communication technology and global positioning system (GPS) technology to provide various mobile communication services such as location information and safe driving in the vehicle 1100. For example, the telematics device may be used to connect the driver, the cloud (e.g., the safety cloud 1106) and / or the surrounding environment to the vehicle 1100. For example, the telematics device may be configured to support 5G NR standard technology (e.g., 5G NR's V2X technology) to support high bandwidth and low latency. For example, the telematics device may be configured to support autonomous driving of the vehicle 1100.

[0189] For example, the gateway 1105 can be used to connect the network inside the vehicle 1100 and the network outside the vehicle, such as the software management cloud 1109 and the security cloud 1106. For example, the software management cloud 1109 can be used to update or manage at least one software required for the driving and management of the vehicle 1100. For example, the software management cloud 1109 can be linked with the in-car security software 1110 installed in the vehicle. For example, the in-car security software 1110 can be used to provide security functions in the vehicle 1100. For example, the in-car security software 1110 can use an encryption key obtained from an external authorized server to encrypt data sent and received through the in-car network to ensure encryption of the in-car network. In various embodiments, the encryption key used by the in-car security software 1110 can be generated based on the identification information of the vehicle (license plate number, vehicle identification number (VIN)) or information specifically given to each user (e.g., user identification information).

[0190] In various embodiments, the gateway 1105 may send the data encrypted by the in-vehicle security software 1110 based on the encryption key to the software management cloud 1109 and / or the security cloud 1106. The software management cloud 1109 and / or the security cloud 1106 may use the decryption key to decrypt the data encrypted by the in-vehicle security software 1110 using the encryption key, thereby identifying which vehicle or user the data is received from. For example, since the decryption key is a unique key corresponding to the encryption key, the software management cloud 1109 and / or the security cloud 1106 may identify the sending subject of the data (e.g., the vehicle or the user) based on the data decrypted by the decryption key.

[0191] For example, the gateway 1105 may be configured to support the in-vehicle safety software 1110 and may be associated with the control device 1000. For example, the gateway 1105 may be associated with the control device 1000 to support a connection between a client device 1107 connected to the safety cloud 1106 and the control device 1000. In another example, the gateway 1105 may be associated with the control device 1000 to support a connection between a third-party cloud 1108 connected to the safety cloud 1106 and the control device 1000. However, this is not limited thereto.

[0192] In various embodiments, the gateway 1105 may be used to connect the software management cloud 1109 that manages the operating software of the vehicle 1100 and the vehicle 1100. For example, the software management cloud 1109 may monitor whether the operating software of the vehicle 1100 needs to be updated, and based on the monitoring result that the operating software of the vehicle 1100 needs to be updated, provide data for updating the operating software of the vehicle 1100 through the gateway 1105. In another example, the software management cloud 1109 may receive a user request for updating the operating software of the vehicle 1100 from the vehicle 1100 through the gateway 1105, and provide data for updating the operating software of the vehicle 1100 based on the reception. However, this is not limited to this.

[0193] Fig.12 1 is a schematic diagram for explaining the operation of the electronic device 101 for training a neural network based on a learning data set. Fig.12 The operation described can be performed by Figures 1a to 11 Executed by an electronic device in.

[0194] refer to Fig.12In operation 1202, according to an embodiment, the electronic device may obtain a learning data set. The electronic device may obtain a learning data set for supervised learning. The learning data may include input data and a ground truth data pair corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives input data of the ground truth data pair. The ground truth data may be obtained by referring to Figures 1a to 12 An electronic device is described for obtaining.

[0195] For example, when training a neural network to recognize an image, the learning data may include information about the image and one or more objects contained in the image. The information may include a category or class of an object that can be recognized by the image. The information may include the position, width, height, and / or size of a visual object corresponding to the object in the image. The learning data set identified by operation 1202 may include multiple learning data pairs. In the example of training a neural network to recognize an image, the learning data set identified by the electronic device may include multiple images and ground truth data corresponding to each of the multiple images.

[0196] Reference Fig.12 In operation 1204, according to one embodiment, the electronic device may train the neural network based on the learning data set. In one embodiment of training the neural network based on supervised learning, the electronic device may input the input data contained in the learning data into the input layer of the neural network. Fig.13 An example of a neural network including the input layer is described. From the output layer of the neural network that receives the input data through the input layer, the electronic device can obtain output data of the neural network corresponding to the input data.

[0197] In one embodiment, the training of operation 1204 may be performed based on the difference between the output data and the ground truth data corresponding to the input data included in the learning data. For example, the electronic device may adjust one or more parameters related to the neural network (e.g., the reference Fig.13 The electronic device may adjust the one or more parameters to reduce the difference. The operation of the electronic device to adjust the one or more parameters may be referred to as tuning the neural network. The electronic device may perform neural network tuning based on the output data by using a function defined for evaluating the performance of the neural network, such as a cost function. The difference between the output data and the ground truth data may be included in an example of the cost function.

[0198] refer to Fig.12 In operation 1206, according to an embodiment, the electronic device may identify whether the neural network trained by operation 1204 outputs valid output data. The output data is valid if the difference (or cost function) between the output data and the ground truth data satisfies the conditions set for using the neural network. For example, when the average value and / or the maximum value of the difference between the output data and the ground truth data is less than or equal to a specified threshold, the electronic device may determine that the neural network outputs valid output data.

[0199] If the neural network does not output valid output data (1206-No), the electronic device may repeatedly perform training of the neural network based on operation 1204. The embodiment is not limited thereto, and the electronic device may repeatedly perform operations 1202 and 1304.

[0200] In a state where valid output data is obtained from the neural network (1206-Yes), the electronic device according to one embodiment may use the trained neural network based on operation 1208. For example, the electronic device may input other input data different from the input data input to the neural network as learning data to the neural network. The electronic device may use the output data obtained from the neural network that receives the other input data as a result of reasoning the other input data based on the neural network.

[0201] Fig.13 is a block diagram of an electronic device 101 according to an embodiment. Fig.13 The electronic device 101 may include Figures 1a to 11 For example, refer to Fig.12 The operation described can be performed by Fig.13 The electronic device 101 and / or Fig.13 Executed by processor 1310.

[0202] Reference Fig.13, the processor 1310 of the electronic device 101 can perform computations related to the neural network 1330 stored in the memory 1320. The processor 1310 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The NPU can be implemented as a chip separate from the CPU, or integrated on the same chip as the CPU in the form of a system on a chip (SoC). The NPU integrated in the CPU may be referred to as a neural core and / or an artificial intelligence (AI) accelerator.

[0203] Reference Fig.13 , the processor 1310 may identify a neural network 1330 stored in the memory 1320. The neural network 1330 may include a combination of an input layer 1332, one or more hidden layers 1334 (or intermediate layers), and an output layer 1336. Each of the above layers (e.g., the input layer 1332, one or more hidden layers 1334, and the output layer 1336) may include a plurality of nodes. The number of hidden layers 1334 may vary depending on the embodiment, and a neural network 1330 including a plurality of hidden layers 1334 may be referred to as a deep neural network. The operation of training the deep neural network may be referred to as deep learning.

[0204] In one embodiment, if the neural network 1330 has a feed forward neural network structure, a first node included in a specific layer may be connected to all second nodes included in other layers before the specific layer. In the memory 1320, the parameters stored for the neural network 1330 may include a weight assigned to a connection between the second node and the first node. In the neural network 1330 having a feed forward neural network structure, the value of the first node may correspond to a weighted sum of values ​​assigned to the second node based on the weight assigned to the connection connecting the second node and the first node.

[0205] In one embodiment, if the neural network 1330 has a convolutional neural network structure, the first node included in a specific layer may correspond to the weighted sum of some second nodes included in other layers before the specific layer. The part of the second nodes corresponding to the first node may be identified by the filter corresponding to the specific layer. In the memory 1320, the parameters stored for the neural network 1330 may include weights representing the filter. The filter may include one or more nodes in the second node for calculating the weighted sum of the first node, and weights corresponding to each of the one or more nodes.

[0206] According to an embodiment, the processor 1310 of the electronic device 101 may train the neural network 1330 using the learning data set 1340 stored in the memory 1320. Based on the learning data set 1340, the processor 1310 may perform a reference Fig.12 The described operations thereby adjust one or more parameters stored in memory 1320 for neural network 1330.

[0207] According to one embodiment, the processor 1310 of the electronic device 101 can perform object detection, object recognition and / or object classification using a neural network 1330 trained based on a learning data set 1340. The processor 1310 can input an image (or video) acquired through a camera 1350 into an input layer 1332 of the neural network 1330. Based on the input layer 1332 into which the image is input, the processor 1310 can sequentially acquire the values ​​of the nodes of each layer contained in the neural network 1330, thereby acquiring a set of values ​​of the nodes of the output layer 1336 (e.g., output data). The output data can be used as a result of reasoning about the information contained in the image using the neural network 1330. The embodiment is not limited thereto, and the processor 1310 can also acquire an image (or video) from an external electronic device connected to the electronic device 101 through a communication circuit 1360 and input it into the neural network 1330.

[0208] In one embodiment, the neural network 1330 trained to process an image can be used to identify an area in the image corresponding to an object (object detection) and / or identify a category of an object represented in the image (object recognition and / or object classification). For example, the electronic device 101 can use the neural network 1330 to segment an area in the image corresponding to the object based on a rectangular shape such as a bounding box. For example, the electronic device 101 can use the neural network 1330 to identify at least one category that matches the object from a plurality of specified categories.

[0209] Fig.14is an example of an operation flow chart performed by an electronic device according to an embodiment. Fig.14 The operation of the electronic device described can be performed by Figure 1a to Figure 1b , Figure 2 Executed by the electronic device 101 and / or the processor 210.

[0210] Reference Fig.14 In operation 1400, according to one embodiment, a processor of an electronic device may acquire an image. The image may be identified in an image sequence (eg, a video) continuously acquired by a camera. The image in operation 1400 may be referred to as an image frame.

[0211] Reference Fig.14 In operation 1402, according to one embodiment, a processor of the electronic device may identify an external object in the acquired image. The external object may be identified based on object recognition. For example, the electronic device may identify the location and / or category of the external object in the image based on a neural network trained for object recognition.

[0212] Reference Fig.14 In operation 1404, according to one embodiment, the processor of the electronic device may classify the external objects identified in operation 1402 that have a possibility of affecting the driving of the vehicle. The processor may compare the possibility of the external objects identified from the image with a specified threshold to perform operation 1404. Based on operation 1404, the processor may identify at least one external object that has a possibility exceeding the specified threshold.

[0213] Reference Fig.14 In operation 1406, according to an embodiment, the processor of the electronic device may assign object recognition information to each external object classified by operation 1404. The object recognition information may include an identifier (or key value) uniquely assigned to the external object. The identifier may be assigned based on a natural number. The object recognition information may include at least one of the position, category and / or the identifier of the external object in the image of operation 1400.

[0214] Reference Fig.14 In operation 1408, according to one embodiment, the processor of the electronic device may start tracking the external object using a linear interpolation method based on the image frame. For example, in operation 1410, according to one embodiment, the processor of the electronic device may identify an external object having a specific identifier (e.g., a natural number i that gradually increases from 1) in the first image frame.

[0215] Reference Fig.14In operation 1412, according to one embodiment, the processor of the electronic device may store information based on the external object identified in operation 1410 and perform labeling. Based on the labeling, the processor may obtain the labeling data of the specific identifier.

[0216] Reference Fig.14 In operation 1414, according to one embodiment, the processor of the electronic device may move to an m-th image frame that is spaced apart from the first image frame by the number of image frames input. In operation 1416, according to one embodiment, the processor of the electronic device may identify an external object having a natural number i identifier of operation 1410 in the m-th image frame. In operation 1418, according to one embodiment, the processor of the electronic device may store information of the external object identified in the m-th image frame and perform annotation.

[0217] Reference Fig.14 In operation 1420, according to an embodiment, the processor of the electronic device may store and label information of an external object that is predicted to be present in the c-th image frame using the external object information stored in the first image frame and the external object information stored in the m-th image frame. The c may be a natural number greater than 1 and less than m. In operation 1420, the external object information acquired for the c-th image frame may correspond to an external object having an identifier of a natural number i.

[0218] Reference Fig.14 In operation 1422, according to one embodiment, the processor of the electronic device may complete the labeling of external objects (e.g., external objects with natural number i identifiers) in all acquired image frames. For example, in operation 1424, the processor may identify whether the labeling of all external objects present in all image frames has been completed. If the labeling of all external objects has not been completed (1424-No), the processor of the electronic device may perform operation 1426.

[0219] Reference Fig.14 In operation 1426, according to an embodiment, the processor of the electronic device may identify other external objects. For example, based on the increase of the natural number i, the processor may identify other external objects that are different from the external object having the natural number i identifier. Fig.14 In operation 1428, according to one embodiment, the processor of the electronic device may identify whether the identifier added based on operation 1426 exceeds the number n of external objects identified based on operation 1402. If the identifier exceeds the number (1428-Yes), the processor may terminate the labeling work. If the identifier does not exceed the number (1428-No), the processor may perform tracking for the external object having the identifier added based on operation 1426.

[0220] Fig.15 FIG. 1 is a schematic diagram for explaining a process of obtaining annotation data related to a region selected by a user input in an image according to an embodiment of the present invention. Fig.15 , reference symbol 1500 denotes an image frame for generating annotation data, and reference symbols 1505 and 1510 denote external objects existing in the image frame 1500, respectively.

[0221] Reference symbol 1550 indicates that, according to an embodiment of the present invention, in order to generate annotation data corresponding to external objects 1505 and 1510, the electronic device displays bounding boxes 1555 and 1560 corresponding to the external objects 1505 and 1510 respectively through user input, and displays pixel values ​​as the horizontal and vertical sizes of the bounding boxes 1555 and 1560. According to an embodiment, the electronic device may generate annotation data corresponding to the external objects 1505 and 1510 based on the reference symbol 1550. Figure 1b The illustrated operation identifies input to bounding boxes 1555 , 1560 .

[0222] Fig.16 is a schematic diagram for illustrating a process of generating annotation data for object recognition learning from an image according to an embodiment of the present invention.

[0223] According to a processor of an electronic device according to an embodiment of the present invention, in order to generate annotation data required for learning to identify objects in an image, an image file 1602 serving as a learning object and a label file 1604 for annotating the image file can be managed as a pair and stored in a storage device.

[0224] Fig.16 Although "jpg" is shown as the format of the image file 1602 to be learned, this is only an embodiment and does not limit the embodiments of the present invention. For example, various video compression formats such as "png" or "gif" can also be used as the format of the image file 1602.

[0225] also, Fig.16The extension of the annotation file 1604 of the image file 1602 is shown as "json", but this is only to illustrate an embodiment of the annotation information for objects in the image file 1602, using a format for expressing data structured in Javascript object syntax, i.e., JSON (Java Script Object Notation), and does not limit the embodiments of the present invention. For example, various dataset formats that can express hierarchical data structures, such as XML (eXtensible Markup Language), csv (comma-separated values), COCO (Common Object in Context), VOC (Visual Object Classe), TFRecord (Tensor Flow Record), etc., can be used as the format of the annotation data, and the format of the annotation data may be different depending on the tool used for annotation.

[0226] Fig.16 In FIG. 1 , reference symbol 1650 shows an example of an image file 1602 as an object for generating annotation data for object detection according to an embodiment of the present invention. The size of the image frame 1650 is 1280 pixels horizontally and 650 pixels vertically. There are a total of 3 objects in the image frame 1650 for which annotation data is to be generated, including 2 pedestrian objects and 1 vehicle object. Specifically, reference symbol 1652 represents a bounding box corresponding to pedestrian object 0, reference symbol 1654 represents a bounding box corresponding to vehicle object 0, and reference symbol 1656 represents a bounding box corresponding to pedestrian object 1. And in FIG. Fig.16 , (X1, Y1) and (X2, Y2) respectively represent the X-axis and Y-axis coordinate points of the position of the bounding box 1652 in the image frame 1650, and respectively represent the upper left corner vertex and the lower right corner vertex of the bounding box 1652.

[0227] In addition, Fig.16 , (X3, Y3) and (X4, Y4) are X-axis and Y-axis coordinate points representing the position of the bounding box 1654 in the image frame 1650, respectively representing the upper left corner vertex and the lower right corner vertex of the bounding box 1654. (X5, Y5) and (X6, Y6) are X-axis and Y-axis coordinate points representing the position of the bounding box 1656 in the image frame 1650, respectively representing the upper left corner vertex and the lower right corner vertex of the bounding box 1656.

[0228] Moreover, in one embodiment of the present invention, referring to Fig.16It can be seen that the object category of the pedestrian object in the bounding box 1652 is designated as "Pedestrian", and the object identifier used to track the object is designated as "0"; the object category of the vehicle object in the bounding box 1654 is designated as "car", and the object identifier used to track the object is designated as "0"; the object category of the pedestrian object in the bounding box 1656 is designated as "pedestrian", and the object identifier used to track the object is designated as "1".

[0229] Fig.16 The object category and object identifier shown in are only an example, and the specific settings may vary according to the user's designation.

[0230] Fig.17 FIG. 1 is a schematic diagram for illustrating the hierarchical structure of generated annotation data according to an embodiment of the present invention. Fig.17 Reference symbol 1702 indicates an example of information about a tool used to generate annotation data in a target file, and reference symbol 1704 indicates an example of a file path corresponding to the annotation data. The tool information of reference symbol 1702 may include the type of tool used for annotation and its version information.

[0231] According to one embodiment of the present invention, the tool for generating labeled data is a software program that can assign specific values ​​to learning data before performing machine learning or deep learning modeling tasks, and may include "labelImg", "CVAT (Computer Vision Annotation Tool)", "LabelMe", "Labelbox", "VoTT", "imglab", "YOLO Mark", "OpenLabeling", "PixelAnnotationTool", "imagetagger", "Alturos.ImageAnnotation", "DeepLabel", "MedTagger", "Turktools", "Pixie", "OpenLabeler", "Anno-Mage", "CATMAID", "makesense.ai", "LOST (Label Object and Save Time)", "annotorious", "sloth", etc. However, this is only one embodiment of the present invention and does not limit the embodiments of the present invention. Specifically, the tool for generating labeled data according to an embodiment of the present invention is a software program that can specify various shapes such as rectangles (Rectangle), polygons (Polygon), lines (Line), points (Point) in an image for labeling for learning object detection and object recognition, and store these bounding box related information as a data structure in a specific format. In addition, reference numeral 1706 denotes an example of information on the width of the file (image file) corresponding to the annotation data, and reference numeral 1708 denotes an example of information on the height of the file (image file) corresponding to the annotation data.

[0232] In addition, reference symbol 1750 is an example of flag information of data as a target of the labeled data. In the embodiment of the present invention, the data as a target of the labeled data is an image file. Since the object recognition rate and object detection rate required for autonomous driving are greatly affected by the weather (cloudy, sunny, rainy, snowy, etc.) and the day and night environment, the flag information is set to sunny day, rainy day, and cloudy day, but this does not limit the present invention.

[0233] In the embodiment of the present invention, the weather of the image frame 1650 as the target of the annotation data is sunny day. Fig.17 In the flags included in the image flag 1750, the sunny day flag is assigned to "True", while the rainy day flag and the cloudy day flag are assigned to "False", respectively. That is, Fig.17 The data shown by reference symbol 1750 in FIG. 1 is only one embodiment, and various marks may be present to improve the object recognition rate and the object detection rate.

[0234] Fig.18 FIG. 1 is a schematic diagram for illustrating the hierarchical structure of external object information in generated annotation data according to an embodiment of the present invention. Fig.18 , reference symbol 1810 corresponds to Fig.16 1652 of the pedestrian object 0 in FIG.

[0235] Referring to the Label parameter of the reference symbol 1810, since the object in the bounding box 1652 is a pedestrian, it is assigned to "Pedestrian", the Tracking ID parameter is assigned to "0", the Shape Type parameter is assigned to "Rectangle", and the Points parameter is assigned to the coordinates ((X1, Y1), (X2, Y2)) of at least two vertices of the bounding box 1652. Fig.18 In the Label Flags parameter, the parameter includes three lower parameters (interpolation, occlusion, and clipping), which respectively indicate whether the object contained in the bounding box 1652 is an interpolation object, whether it is occluded by other objects, and whether it is a partially clipped object. In the Label Flags of reference symbol 1810, the "interpolation" parameter is assigned to "True", and the "occlusion" parameter and the "clipping" parameter are both assigned to "False".

[0236] The Label parameter of reference symbol 1830 is assigned to "Car" because the object in the bounding box 1654 is a vehicle, the Tracking ID parameter is assigned to "0", the Shape Type parameter is assigned to "Rectangle", and the Points parameter is assigned to the coordinate values ​​((X3, Y3), (X4, Y4)) of at least two vertices of the bounding box 1654. In addition, the "Interpolation" parameter, the "Occlusion" parameter, and the "Clipping" parameter in the Label Flags of reference symbol 1830 are all assigned to "False".

[0237] The Label parameter of reference symbol 1850 is assigned to "Pedestrian" because the object in the bounding box 1656 is a pedestrian. In order to distinguish other pedestrian objects in the bounding box 1656, the Tracking ID parameter is assigned to "1", the ShapeType parameter is assigned to "Rectangle", and the Points parameter is assigned to the coordinate values ​​of at least two vertices of the bounding box 1656 ((X5, Y5), (X6, Y6)). In addition, the "Interpolation" parameter, the "Occlusion" parameter, and the "Clipping" parameter in the Label Flags of reference symbol 1850 are all assigned to "False".

[0238] Fig.18 The information of external objects in the labeling data shown in the hierarchical structure is shown as an example to help understand the present invention, and the external object categories may include "Bus", "Truck", "Pick-up Truck", "Sedan", "SUV (Sports Utility Vehicle)", "Bike", "Scooter", "Motor-Cycle" and other external objects that need to be identified for vehicle autonomous driving.

[0239] also, Fig.18 In the example, the bounding box corresponding to the object to be annotated is assigned as a "Rectangle", so the Points representing its position coordinates are assigned as two pairs of X-axis coordinates and Y-axis coordinate values, but if the Shape Type is "Polygon", the X-axis coordinates and Y-axis coordinate value pairs of the points created by the user along the appearance of the target object to be annotated will be stored in the order in which the points were created by the user. According to an embodiment of the present invention, when the Shape Type is "Polygon", the coordinate examples assigned to the Points parameter will be referred to below. Fig.19 Provide explanation.

[0240] Fig.19 1 is a diagram of an example of coordinates assigned to Points Parameter when Shape Type is "Polygon" according to an embodiment of the present invention. Fig.19When the "Shape Type" of the annotation data of the vehicle 1950, which is the external object in the image frame and the annotation target, is "Polygon", and the user has created 20 points from 1 to 20, as shown by reference symbol 1970, pairs of X-axis coordinates and Y-axis coordinate values ​​(X1, Y1) to (X20, Y20) corresponding to the 20 points from 1 to 20 are assigned.

[0241] The annotation data described in the present invention may be stored in a storage device of a user's local computer or a storage device of a cloud server.

[0242] Fig. 20 is a diagram for illustrating an example of dividing the generated labeled data into a training set, a validation set and / or a test set in order to achieve effective training of a neural network according to an embodiment of the present invention.

[0243] Reference Fig. 20 According to an embodiment of the present invention, the electronic device can analyze the image flag (Image Flag) 2010 of the labeled data (Labeled Data) 2005 and classify the labeled data according to the analyzed image flag.

[0244] According to an embodiment of the present invention, the processor of the electronic device can classify the data marked as reference symbol 2020 according to the weather-related symbols or according to the time period according to the image symbol. Fig. 20 In the example, the labeled data 2010 can be divided into sunny day, cloudy day, snowy day, rainy day and foggy day according to weather related signs.

[0245] In addition, the processor of the electronic device according to an embodiment of the present invention can classify the annotation data 2005 into morning, afternoon, night, sunset, and sunrise according to the image mark and the time period mark.

[0246] In addition, the processor of the electronic device according to an embodiment of the present invention can compare the amount of the labeled data classified as reference symbol 2020 with a value preset by the user, and determine the division method of the labeled data 2005 into the training set 2042, the validation set 2044 and / or the test set 2046 according to the comparison result. At this time, the preset value used to determine the division method of the labeled data is a value determined in advance by the user, which can be determined by a previous experience value or experimental value used to determine whether the labeled data is sufficient to generate the validation set 2044.

[0247] Reference symbol 2040 shows an example of a manner in which the processor of the electronic device according to an embodiment of the present invention divides the labeled data 2005 into a training set 2042, a validation set 2044, and / or a test set 2046 when the amount of the labeled data 2005 is greater than the preset value. At this time, in reference symbol 2040, the processor of the electronic device can divide the labeled data into the training set 2042, the validation set 2044, and / or the test set 2046 according to the ratio preset by the user.

[0248] On the contrary, reference symbol 2060 shows an example of the manner in which the processor of the electronic device according to an embodiment of the present invention divides the labeled data 2005 into a training set 2042 and a test set 2046 when the amount of the labeled data 2005 is less than or equal to the preset value. At this time, in reference symbol 2060, the processor of the electronic device divides the labeled data into a training set 2042 and a test set 2046 according to a ratio preset by the user, and in order to verify the reliability of the training set 2042, K-fold cross validation, one of the algorithms 2065 for verifying the reliability of a neural network, is used.

[0249] As described above, an electronic device according to an embodiment may include a memory and a processor. The processor may identify a first position related to an external object from a first image at a first time point in a plurality of images for a video in the memory and in the plurality of images. The processor may identify a second position related to the external object from a second image at a second time point after the first time point in the plurality of images based on the first position in the first image. The processor may acquire one or more third positions related to the external object corresponding to one or more third images included in a time period between the first time point and the second time point based on the first position and the second position. The processor may store the first position, the one or more third positions, and the second position as annotation information representing the movement of the external object identified in the time period of the video.

[0250] For example, the processor can interpolate the first coordinate representing the first position in the first image at the first time point and the second coordinate representing the second position in the second image at the second time point using the length between the first time point and the second time point to obtain the one or more third positions.

[0251] For example, the processor may interpolate the first coordinate and the second coordinate based on the time points of the one or more third images within the time period to obtain the one or more third positions.

[0252] For example, the processor may compare one or more feature points contained in the portion of the first image having the first position corresponding to the external object and one or more feature points contained in the second image corresponding to the external object, thereby identifying the second position in the second image.

[0253] For example, the processor may change the third position based on identifying the second position in the second image at the second time point after a critical interval after the first time point, using at least one feature point included in the one or more third images.

[0254] For example, the processor may input the first image and the second image into a model for identifying the external object, thereby identifying the first position and the second position of the external object captured within the time period.

[0255] For example, the electronic device may further include a display. The processor may display a screen for playing the video on the display. The processor may display a visual object for indicating the position of the external object superimposed on the image displayed on the screen based on the annotation information, while displaying one of the multiple images on the screen based on an input for indicating the playing of the video.

[0256] For example, the processor may recognize an input for indicating movement of the visual object in the state where the screen displays one of the one or more third images. Based on the input, the processor adjusts the position of the external object corresponding to another image different from the image displayed on the screen in the one or more third images according to the position of the visual object moved by the input.

[0257] According to the method of the electronic device of one embodiment as described above, the step of identifying a first position related to an external object from a first image at a first time point in a plurality of images for a video in a memory of the electronic device and the plurality of images. The method may include the step of identifying a second position related to the external object from a second image at a second time point after the first time point in the plurality of images based on the first position in the first image. The method may include the step of acquiring one or more third positions related to the external object corresponding to one or more third images included in a time period between the first time point and the second time point based on the first position and the second position. The method may include the step of storing the first position, the one or more third positions and the second position as annotation information representing the movement of the external object identified in the time period of the video.

[0258] For example, the acquisition step may include the step of interpolating the first coordinate representing the first position in the first image at the first time point and the second coordinate representing the second position in the second image at the second time point using the length between the first time point and the second time point to obtain the one or more third positions.

[0259] For example, the step of acquiring may include the step of interpolating the first coordinate and the second coordinate based on the time points of the one or more third images within the time period to acquire the one or more third positions.

[0260] For example, the step of identifying the second position may include the step of comparing one or more feature points contained in the portion of the first image having the first position corresponding to the external object, thereby identifying the second position in the second image.

[0261] For example, the acquisition step may include a step of changing the third position based on identifying the second position in the second image at the second time point after the critical interval after the first time point, using at least one feature point contained in the one or more third images.

[0262] For example, the step of identifying the second position may include inputting the first image and the second image into a model for identifying the external object, thereby identifying the first position and the second position of the external object captured within the time period.

[0263] For example, the method may include the step of displaying a screen for playing the video on the display of the electronic device. The method may include the step of displaying a visual object for indicating the position of the external object superimposed on the image displayed in the screen based on the annotation information, while one of the multiple images is displayed in the screen based on an input for indicating the playing of the video.

[0264] For example, the method may include a step of recognizing an input for indicating the movement of the visual object in the state where the screen displays one of the one or more third images. The method may also include a step of adjusting, based on the input, the position of the external object corresponding to another image in the one or more third images that is different from the image displayed on the screen, according to the position of the visual object moved by the input.

[0265] As described above, an electronic device according to an embodiment may include a display, a memory, and a processor. The processor may, in a state where the display displays a first image of a video stored in the memory, recognize a first input indicating a selection of a first position associated with an external object in the first image. The processor may, based on the first input, perform a first type of computation for identifying the external object so as to identify a second position associated with the external object in a second image after a time period starting from a time point of the first image from among multiple images of the video. The processor may perform a second type of computation for interpolating the first position and the second position so as to obtain a third position associated with the external object from one or more third images included in the time period. The processor may, in response to a second input indicating the playback of at least a portion of the video included in the time period, display the first image, one of the one or more third images, and the second image on the display, and superimpose a visual object corresponding to one of the first position, the third position, or the second position on the image displayed on the display.

[0266] For example, the processor may repeatedly perform the first type of calculation for identifying the external object based on one or more feature points within each of the time periods starting from the first time point of the first image.

[0267] For example, the processor may perform the second type of calculation for acquiring the third position based on the first position, the second position, and time points of the one or more third images within the time period.

[0268] For example, the processor may store the first position, the third position, and the second position in the memory as annotation information corresponding to the video.

[0269] For example, the processor may associate the data representing the time point of the first image with the first position and store the data in the annotation information.

[0270] According to the method of the electronic device of one embodiment as described above, the method may include the step of recognizing a first input indicating the selection of a first position related to an external object in a first image of a video stored in a memory of the electronic device in a state where the display of the electronic device displays a first image. The method may include the step of performing a first type of calculation for identifying the external object based on the first input, so as to identify a second position related to the external object in a second image after a time period starting from a time point of the first image from among multiple images of the video. The method may include the step of performing a second type of calculation for interpolating the first position and the second position, so as to obtain a third position related to the external object from one or more third images included in the time period. The method may include the step of displaying the first image, one of the one or more third images and the second image on the display in response to a second input indicating the playback of at least a portion of the video included in the time period, and superimposing and displaying a visual object corresponding to one of the first position, the third position or the second position on the image displayed on the display.

[0271] For example, the step of identifying the second position may include the step of repeatedly performing the calculation of the first type for identifying the external object based on one or more feature points in each of the time periods starting from the first time point of the first image.

[0272] For example, the step of acquiring the third position may include the step of performing the second type of calculation for acquiring the third position based on the first position, the second position and time points of the one or more third images within the time period.

[0273] For example, the method may further include the step of storing the first position, the third position, and the second position in the memory as annotation information corresponding to the video.

[0274] For example, the storing step may include a step of associating data representing the time point of the first image with the first position and storing the data in the annotation information.

[0275] The device described above can be implemented by hardware components, software components and / or a combination of hardware components and software components. For example, the device and components described in the embodiment can be implemented by a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (digital signal processor), a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device that can execute and respond to instructions (instruction) and more than one general-purpose computer or a special-purpose computer. The processing device can execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device can access, store, operate, process and generate data in response to the execution of the software. For ease of understanding, although it is mentioned in the description that only one processing device is used, it will be understood by those of ordinary skill in the art that the processing device may include multiple processing elements (processing element) and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, other processing configurations (processing configuration) such as parallel processors (parallelprocessor) can also be used.

[0276] Software may include a computer program, code, instructions, or any combination of these elements that can cause a processing device to operate as intended, or command a processing device individually or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by or provide instructions or data to a processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0277] The method according to the embodiment can be implemented in the form of program instructions that can be executed by various computer means, and recorded on a computer-readable medium. At this time, the medium can continuously store programs that can be executed on the computer, or temporarily store programs for execution or downloading. The medium can also be a recording or storage means in various forms of a single or multiple hardware combination, not limited to a medium directly connected to a computer system, but also a medium distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and tapes, optical storage media such as CD-ROMs and DVDs, magneto-optical media such as optical floppy disks (Flopticaldisk), and means for storing program instructions such as ROM, RAM, and flash memory. In addition, examples of other media may also include recording media and even storage media managed by application stores that distribute application programs or other various websites, servers, etc. that supply or distribute software.

[0278] Although the embodiments are described above through limited examples and drawings, a person skilled in the art can make various modifications and variations based on the above description. For example, the described techniques can be performed in a different order from the described methods, and / or the described systems, structures, devices, circuits and other components can be combined or combined in a different form from the described methods, or replaced by other components or equivalents, and even so, appropriate results can be achieved.

[0279] Accordingly, other implementations, other embodiments, and their equivalents are also within the scope of the claims.

Claims

1. An electronic device, include: Memory; as well as processor; Wherein, the processor identifying, from a plurality of images in the memory for the video and a first image at a first point in time in the plurality of images, a first location associated with an external object; identifying, based on the first position in the first image, a second position associated with the external object in a second image at a second time point after the first time point in the plurality of images; Based on the first position and the second position, acquiring one or more third positions corresponding to one or more third images included in a time period between the first time point and the second time point and related to the external object; and The first position, the one or more third positions, and the second position are stored as annotation information representing the movement of the external object identified during the time period of the video.

2. The electronic device according to claim 1, It is characterized in that the processor, The first coordinates representing the first position in the first image at the first time point and the second coordinates representing the second position in the second image at the second time point are interpolated using the length between the first time point and the second time point to obtain the one or more third positions.

3. The electronic device as claimed in claim 2, It is characterized in that the processor, Based on time points of the one or more third images within the time period, the first coordinate and the second coordinate are interpolated to obtain the one or more third positions.

4. The electronic device as claimed in claim 2, It is characterized in that the processor, The second position is identified in the second image by comparing one or more feature points included in the portion of the first image having the first position corresponding to the external object and one or more feature points included in the second image corresponding to the external object.

5. The electronic device as claimed in claim 1, It is characterized in that the processor, Based on identifying the second position in the second image at the second time point after a critical interval after the first time point, the third position is changed using at least one feature point included in the one or more third images.

6. The electronic device as claimed in claim 1, It is characterized in that the processor, The first image and the second image are input into a model for identifying the external object, thereby identifying the first position and the second position of the external object captured within the time period.

7. The electronic device as claimed in claim 1, It is characterized in that Also includes a display, the processor, Displaying a picture for playing the video on the display; In a state where one of the plurality of images is displayed in the screen based on an input indicating playback of the video, a visual object indicating a position of the external object is superimposed and displayed on the image displayed in the screen based on the annotation information.

8. The electronic device as claimed in claim 7, It is characterized in that the processor, In the state where the screen displays one of the one or more third images, recognizing an input for indicating movement of the visual object; Based on the input, the position of the external object corresponding to another image different from the image displayed on the screen in the one or more third images is adjusted according to the position of the visual object moved by the input.

9. A method of an electronic device, comprising the steps of: identifying, from a plurality of images for the video in a memory of the electronic device and a first image at a first point in time among the plurality of images, a first location associated with an external object; identifying, based on the first position in the first image, a second position associated with the external object in a second image at a second time point after the first time point in the plurality of images; Based on the first position and the second position, acquiring one or more third positions corresponding to one or more third images included in a time period between the first time point and the second time point and related to the external object; and The first position, the one or more third positions, and the second position are stored as annotation information representing the movement of the external object identified during the time period of the video.

10. The method of the electronic device as claimed in claim 9, It is characterized in that The step of obtaining comprises: The first coordinates representing the first position in the first image at the first time point and the second coordinates representing the second position in the second image at the second time point are interpolated using the length between the first time point and the second time point to obtain the one or more third positions.

11. The method of the electronic device as claimed in claim 10, It is characterized in that The step of obtaining comprises: Based on time points of the one or more third images within the time period, the first coordinate and the second coordinate are interpolated to obtain the one or more third positions.

12. The method of the electronic device as claimed in claim 10, It is characterized in that The step of identifying the second position comprises: One or more feature points contained in a portion of the first image having the first position corresponding to the external object are compared to identify the second position in the second image.

13. The method of the electronic device as claimed in claim 9, It is characterized in that The step of obtaining comprises: Based on identifying the second position in the second image at the second time point after a critical interval after the first time point, the third position is changed using at least one feature point included in the one or more third images.

14. The method of the electronic device as claimed in claim 9, It is characterized in that The step of identifying the second position comprises: The first image and the second image are input into a model for identifying the external object, thereby identifying the first position and the second position of the external object captured within the time period.

15. The method of the electronic device as claimed in claim 9, It is characterized in that The following steps are also included: Displaying a picture for playing the video on a display of the electronic device; In a state where one of the plurality of images is displayed in the screen based on an input indicating playback of the video, a visual object indicating a position of the external object is superimposed and displayed on the image displayed in the screen based on the annotation information.

16. The method of the electronic device as claimed in claim 15, It is characterized in that The following steps are also included: In the state where the screen displays one of the one or more third images, recognizing an input for indicating movement of the visual object; Based on the input, the position of the external object corresponding to another image different from the image displayed on the screen in the one or more third images is adjusted according to the position of the visual object moved by the input.

17. An electronic device, include: monitor; Memory; as well as processor; Wherein, the processor In a state where the display displays a first image of a video stored in the memory, recognizing a first input indicating selection of a first position associated with an external object in the first image; performing a first type of computation for identifying the external object based on the first input so as to identify, from among a plurality of images of the video, a second location associated with the external object in a second image that is a time period later than a time point of the first image; performing a second type of calculation for interpolating the first position and the second position to obtain a third position associated with the external object from one or more third images included in the time period; and In response to a second input indicating playing at least a portion of the video contained in the time period, the first image, the one or more third images, and one of the second images are displayed on the display, and a visual object corresponding to one of the first position, the third position, or the second position is superimposed on the image displayed on the display.

18. The electronic device as claimed in claim 17, It is characterized in that the processor, The first type of calculation for identifying the external object based on one or more feature points is repeatedly performed in each of the time periods starting from the first time point of the first image.

19. The electronic device according to claim 17, It is characterized in that the processor, The calculation of the second type for obtaining the third position is performed based on the first position, the second position and the time points of the one or more third images within the time period.

20. The electronic device as claimed in claim 17, It is characterized in that the processor, The first position, the third position, and the second position are stored in the memory as annotation information corresponding to the video.

Images