Electronic device, method, and computer readable storage medium for determining type of video sequence using artificial neural network

Abstract

According to one embodiment, a computer-readable storage medium may store one or more programs including instructions that cause the electronic device to receive information about a video sequence from an external electronic device through the communication circuit when executed by at least one processor of an electronic device having a communication circuit, identify a part of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence using a trained neural network, and determine the type of the part of the video sequence as a first type corresponding to the designated scene among a plurality of types or a second type distinguished from the first type among the plurality of types, depending on whether the designated visual object is included within a designated location within each of the plurality of images constituting the part of the video sequence using the trained neural network.

Description

Technology Field

[0001] The various embodiments described below relate to an electronic device, a method, and a computer-readable storage medium for determining the type of a video sequence using an artificial neural network. Background Technology

[0003] An artificial neural network is a statistical learning algorithm inspired by machine learning and biological neural networks. An artificial neural network can refer to a model that possesses problem-solving capabilities by modifying the strength of synaptic connections through the learning of nodes that form a network via synaptic connections. Such artificial neural networks can be used to recognize multiple scenes indicated by a video sequence composed of multiple images. The problem to be solved

[0005] An electronic device may recognize a plurality of scenes indicated by a video sequence using an artificial neural network, and based on the result of the recognition, divide the video sequence into a plurality of partial video sequences, each comprising the recognized plurality of scenes. The plurality of partial video sequences are classified into a plurality of types to represent each of the plurality of scenes and may be provided according to the plurality of types.

[0006] Meanwhile, the above video sequence may include partial video sequences that indicate the same scene but are acquired in different contexts. Since these partial video sequences are acquired in different contexts, they may need to be provided in different types.

[0007] The technical problems to be solved in this document are not limited to those described above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this invention belongs from the description below. means of solving the problem

[0009] According to one embodiment, a computer-readable storage medium may store one or more programs including instructions that cause the electronic device to receive information about a video sequence from an external electronic device through the communication circuit when executed by at least one processor of an electronic device having a communication circuit, identify a part of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence using a trained neural network, and determine the type of the part of the video sequence as a first type corresponding to the designated scene among a plurality of types or a second type distinguished from the first type among the plurality of types, depending on whether the designated visual object is included within a designated location within each of the plurality of images constituting the part of the video sequence using the trained neural network.

[0010] A method executed within an electronic device having a communication circuit according to one embodiment may include: receiving information about a video sequence from an external electronic device through the communication circuit; identifying a part of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence using a trained neural network; and determining, using the trained neural network, the type of the part of the video sequence as a first type corresponding to the designated scene among a plurality of types or a second type distinguished from the first type among a plurality of types, depending on whether the designated visual object is included within a designated position within each of a plurality of images constituting the part of the video sequence.

[0011] According to one embodiment, an electronic device may include a memory configured to store instructions, a communication circuit, and a processor configured to operatively combine with the memory and the communication circuit to receive information about a video sequence from an external electronic device through the communication circuit, identify a portion of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence using a trained neural network, and determine the type of the portion of the video sequence as a first type corresponding to the designated scene among a plurality of types or a second type distinguished from the first type among a plurality of types, depending on whether the designated visual object is included within a designated position within each of a plurality of images constituting the portion of the video sequence using the trained neural network. Effects of the invention

[0013] An electronic device, method, and computer-readable storage medium according to one embodiment can enhance the quality of a service provided through a partial video sequence by recognizing, based on a visual object, a situation in which a partial video sequence is provided or a situation in which a partial video sequence is acquired.

[0014] The effects obtainable from the present disclosure are not limited to those described above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below. Brief explanation of the drawing

[0016] FIG. 1 illustrates an example of an environment including an electronic device according to one embodiment. FIG. 2 is a simplified block diagram of an electronic device according to one embodiment. FIG. 3a illustrates an example of an image included in at least some of a plurality of images within a video sequence. FIG. 3b illustrates another example of an image included in at least some of a plurality of images within a video sequence. Figure 4a illustrates an example of a specified visual object within an image. Figure 4b illustrates another example of a specified visual object within an image. Figure 4c illustrates another example of a specified visual object within an image. FIG. 5a is a flowchart illustrating a method of configuring training data according to one embodiment. FIG. 5b illustrates an example of labeling a specified location of a specified visual object. Figure 5c illustrates another example of labeling a specified location of a specified visual object. FIG. 6 is a flowchart illustrating a method for determining the type of part of a video sequence according to one embodiment. FIG. 7 is a flowchart illustrating another method for determining the type of part of a video sequence according to one embodiment. Specific details for implementing the invention

[0017] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a server, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0018] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another corresponding component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0019] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed as a whole, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0020] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium readable by a machine (e.g., an electronic device (105)). For example, a processor of the machine (e.g., an electronic device (105)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0021] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM , or App Store TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0022] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0024] FIG. 1 illustrates an example of an environment including an electronic device according to one embodiment.

[0025] Referring to FIG. 1, the environment (100) may include an electronic device (101), an electronic device (102), and an electronic device (103).

[0026] In one embodiment, the electronic device (101) may be a device for acquiring a video sequence. For example, if the video sequence is a video sequence acquired by filming a sports game through a camera, the electronic device (101) may be a device used by a service provider that provides a broadcasting service using the video sequence acquired through the camera. In one embodiment, the broadcasting service may be provided live.

[0027] In one embodiment, the electronic device (101) may be a device that obtains the video sequence obtained through the camera from the service provider providing the broadcasting service and relays the obtained video sequence to another electronic device.

[0028] In one embodiment, the electronic device (101) may provide the video sequence to the electronic device (102). For example, the electronic device (101) may provide the video sequence to the electronic device (102) through a communication circuit of the electronic device (101). For example, the electronic device (101) may provide the video sequence to the electronic device (102) through a wired communication path. As another example, the electronic device (101) may provide the video sequence to the electronic device (102) through a wireless communication path. However, it is not limited thereto. Meanwhile, the electronic device (102) may obtain the video sequence from the electronic device (101).

[0029] In one embodiment, the electronic device (102) may be a device for acquiring a plurality of partial video sequences divided from the video sequence acquired from the electronic device (101). In one embodiment, the electronic device (102) may recognize a plurality of scenes indicated by the video sequence by analyzing the video sequence using a trained neural network (or model) and acquire the plurality of partial video sequences each containing the plurality of scenes. In one embodiment, the plurality of partial video sequences may be classified into a plurality of types corresponding to each of the plurality of scenes. For example, the electronic device (102) may classify the plurality of partial video sequences into the plurality of types and provide each of the plurality of partial video sequences classified into the plurality of types to the electronic device (103).

[0030] In one embodiment, among the plurality of partial video sequences, the first partial video sequence indicates the same scene as indicated by the second partial video sequence among the plurality of partial video sequences, but the first partial video sequence and the second partial video sequence may be partial video sequences that must be provided in different situations. For example, if the first partial video sequence indicates or includes a first scene acquired through a camera, and the second partial video sequence indicates or includes a replay scene (or highlight scene) acquired through editing of the first scene, the first partial video sequence and the second video sequence may include the same scene but must be provided in different situations. For example, if the first partial video sequence and the second partial video sequence are provided as a single video sequence within the electronic device (103), inconvenience may be caused to the user of the electronic device (103) because the user watches the single video sequence that repeatedly provides the same scene. As another example, if the first partial video sequence and the second partial video sequence are provided as independent video sequences within a user interface provided within the electronic device (103), inconvenience to the user may be caused because the user of the electronic device (103) may view the same scene twice. Additionally, if the first partial video sequence and the second partial video sequence are provided as independent video sequences, efficiency in terms of content storage may be reduced.

[0031] An electronic device (102) according to one embodiment may perform an operation to distinguish between the situation during acquisition of a scene included in the first partial video sequence and the situation during acquisition of a scene included in the second partial video sequence in order to reduce the inconvenience (or inefficiency) caused by providing a video sequence. For example, the electronic device (102) may distinguish between the situation during acquisition of a scene included in the first partial video sequence and the situation during acquisition of a scene included in the second partial video sequence by using the trained neural network to identify whether a designated visual object is included within a designated location within a plurality of images constituting each of the first partial video sequence and the second partial video sequence. A description of the designated visual object and the designated location will be provided later.

[0032] Meanwhile, in one embodiment, the electronic device (102) may provide at least some of the plurality of partial sequences divided from the video sequence to the electronic device (103). The electronic device (103) may obtain at least some of the plurality of partial sequences from the electronic device (102).

[0033] In one embodiment, the electronic device (103) may be a device that displays at least some of the plurality of partial sequences obtained from the electronic device (102) through a display of the electronic device (103). For example, the electronic device (103) may be a user device used by a user who views at least some of the plurality of partial sequences. For example, the electronic device (103) may be a desktop computer, a laptop computer, a tablet PC (personal computer), a smart television, or a smartphone. However, it is not limited thereto.

[0034] For example, the electronic device (103) may display executable objects for accessing or playing at least some of the plurality of partial sequences within a user interface of an application associated with the electronic device (102). The executable objects may be composed of a plurality of visual objects, each including a thumbnail image for representing at least some of the plurality of partial sequences. However, they are not limited thereto.

[0035] As described above, an electronic device (102) according to one embodiment can enhance the user experience for the plurality of partial video sequences and enhance the service quality associated with the plurality of partial video sequences by classifying the video sequence obtained from the electronic device (101) into a plurality of partial video sequences based on scenes included in the video sequence identified using the neural network as well as situations during the acquisition of the scenes identified using the neural network.

[0037] FIG. 2 is a simplified block diagram of an electronic device according to one embodiment.

[0038] FIG. 3a illustrates an example of an image included in at least some of a plurality of images within a video sequence.

[0039] FIG. 3b illustrates another example of an image included in at least some of a plurality of images within a video sequence.

[0040] Figure 4a illustrates an example of a specified visual object within an image.

[0041] Figure 4b illustrates another example of a specified visual object within an image.

[0042] Figure 4c illustrates another example of a specified visual object within an image.

[0043] FIG. 5a is a flowchart illustrating a method of configuring training data according to one embodiment.

[0044] FIG. 5b illustrates an example of labeling a specified location of a specified visual object.

[0045] Figure 5c illustrates another example of labeling a specified location of a specified visual object.

[0046] FIG. 6 is a flowchart illustrating a method for determining the type of part of a video sequence according to one embodiment.

[0047] FIG. 7 is a flowchart illustrating another method for determining the type of part of a video sequence according to one embodiment.

[0049] Referring to FIG. 2, the electronic device (200) may be an example of the electronic device (102) illustrated in FIG. 1. The electronic device (200) may include a processor (202), memory (204), storage device (206), a high-speed controller (208) (e.g., northbridge, MCH (main controller hub)), and a low-speed controller (212) (e.g., southbridge, ICH (I / O (input / output) controller hub)). Within the electronic device (200), each of the processor (202), memory (204), storage device (206), high-speed controller (208), and low-speed controller (212) may be interconnected using various buses.

[0050] For example, the processor (202) may process instructions for execution within the electronic device (200) to display graphic information for a graphical user interface (GUI) on an external input / output device, such as a display (216) connected to a high-speed controller (208). The instructions may be contained in memory (204) or storage device (206). The instructions may cause the electronic device (200) to perform one or more of the operations described above and / or one or more of the operations described below when executed by the processor (202). According to embodiments, the processor (202) may be composed of a plurality of processors including a communication processor and a graphical processing unit (GPU).

[0051] For example, the memory (204) can store information within the electronic device (200). For example, the memory (204) may be a volatile memory unit or units. For another example, the memory (204) may be a non-volatile memory unit or units. For yet another example, the memory (204) may be another form of computer-readable medium, such as a magnetic or optical disk.

[0052] For example, the storage device (206) can provide mass storage space to the electronic device (200). For example, the storage device (206) may be a computer-readable medium, such as a hard disk device, an optical disk device, a flash memory, a solid-state memory device, or an array of devices within a storage area network (SAN).

[0053] For example, the high-speed controller (208) manages bandwidth-intensive operations for the electronic device (200), while the low-speed controller (212) can manage low bandwidth-intensive operations for the electronic device (200). For example, the high-speed controller (208) is coupled to the memory (204) and coupled to the display (216) via a GPU or accelerator, while the low-speed controller (212) is coupled to the storage device (206) and can be coupled to various communication ports (e.g., USB (universal serial bus), Bluetooth, Ethernet, wireless Ethernet) or communication circuits for communication with external electronic devices (e.g., keyboard, transducer, scanner, or network device (e.g., switch or router)).

[0054] In one embodiment, the processor (202) may acquire a video sequence from the electronic device (200) through the communication circuit. The video sequence may indicate or include a plurality of scenes acquired through a camera.

[0055] In one embodiment, the video sequence may be a video sequence obtained by capturing a sports game (e.g., a baseball game) through the camera. In one embodiment, the video sequence may include not only scenes of the sports game obtained through the camera, but also information for guiding the state of the sports game. For example, the video sequence may include the information superimposed on the scenes of the sports game. For example, referring to FIG. 3a, the video sequence may include an image (300) containing one scene (305) of the scenes of the sports game. The image (300) may include information (310) superimposed on the scene (305). For example, the information (310) may be used to guide the state of the sports game. For example, if the sports game is a baseball game, the information (310) may include at least some of the following: data for indicating what inning corresponds to the scene (305); data for indicating which sports teams the scene (305) is a game between; data for indicating the scoring status of the sports teams in the scene (305); data for indicating the ball count in the scene (305); data for indicating the runner's advancement status in the scene (305); data for indicating the batter's season record in the scene (305); data for indicating the batter's record in the current game in the scene (305); or data for indicating the number of pitches thrown by the pitcher in the current game in the scene (305). However, it is not limited thereto.

[0056] Referring again to FIG. 2, in one embodiment, the video sequence may include scenes corresponding to advertising content provided during the break time of the sports game. In one embodiment, the video sequence may include not only scenes corresponding to the advertising content but also information for guiding the state of the sports game superimposed on the advertising content. For example, referring to FIG. 3b, the video sequence may include an image (315) corresponding to at least a portion of the advertising content. The image (315) may include a scene (320) constituting at least a portion of the advertising content and information (325) superimposed on the scene (320). For example, the information (325) may be used to guide the state of the sports game during the break time. For example, if the sports game is a baseball game, the information (325) may include at least some of the following: data for indicating the scoring status of the baseball game immediately before the break time, data for indicating the caster or commentator broadcasting the baseball game, or data for indicating who the service provider is that provides the broadcast of the baseball game. However, it is not limited thereto.

[0057] Referring again to FIG. 2, the processor (202) may perform an operation to identify each of the situations while acquiring each of the plurality of partial video sequences divided from the video sequences, which include scenes of the sports game, the information (e.g., information (310)) superimposed on at least some of the scenes of the sports game, scenes corresponding to advertising content provided during the break time of the sports game, and the information (e.g., information (325)) superimposed on at least some of the scenes corresponding to the advertising content. In one embodiment, the processor (202) may perform the operation using the trained neural network. In one embodiment, the processor (202) may perform the operation using the neural network that has been trained to identify the information (e.g., information (310) and information (325)). The information may be referred to as a designated visual object in that it is used to identify the situations. In one embodiment, the processor (202) may use the neural network to identify whether the designated visual object is included in each of the plurality of partial video sequences in order to identify each of the situations. In one embodiment, the designated visual object may be superimposed on a scene of a sports game (e.g., scene (305)), as in the information (310) of FIG. 3a.

[0058] The designated visual object may be configured with a plurality of visual elements. The plurality of visual elements may be configured within the designated visual object in various arrangements. For example, referring to FIG. 4a, if the sports game is a baseball game, the designated visual object (410) may include a first visual element (410-1), a second visual element (410-2) next to the first visual element (410-1), a third visual element (410-3) next to the second visual element (410-2), a fourth visual element (410-4) next to the third visual element (410-3), a fifth visual element (410-5) next to the fourth visual element (410-4), and a sixth visual element (410-6) below the first visual element (410-1) through the fifth visual element (410-5). As another example, the designated visual object (420), unlike the designated visual object (410), may include a first visual element (420-1), a second visual element (420-2) below the first visual element (420-1), a third visual element (420-3) placed below the second visual element (420-2), a fourth visual element (420-4) placed below the second visual element (420-2) and next to the third visual element (420-3), and a fifth visual element (420-5) placed below the third visual element (420-3) to the fourth visual element (420-4). As another example, the designated visual object (430), unlike the designated visual object (410) and the designated visual object (420), may include a first visual element (430-1), a second visual element (430-2) next to the first visual element (430-1), a third visual element (430-3) next to the second visual element (430-2), a fourth visual element (430-4) next to the third visual element (430-3), and a fifth visual element (430-5) next to the fourth visual element (430-4).As another example, the designated visual object (440), unlike the designated visual object (410) to the designated visual object (430), may include a first visual element (440-1), a second visual element (440-2) placed below the first visual element (440-1), a third visual element (440-3) placed below the first visual element (440-1) and next to the second visual element (440-2), a fourth visual element (440-4) placed below the first visual element (440-1) and next to the third visual element (440-3), and a fifth visual element (440-5) placed below the first visual element (440-1) and next to the fourth visual element (440-4). However, it is not limited thereto.

[0059] Referring again to FIG. 2, the processor (202) may use the neural network to identify whether the designated visual object includes a designated visual element in order to identify each of the above situations. For example, the designated visual element may be a visual element indicating that the scene of the sports game indicated by each of the plurality of partial video sequences corresponds to a replay scene. For example, referring to FIG. 4b, the processor (202) may identify whether the designated visual object (450), which is the designated visual object, includes a visual element (450-1) indicating that the scene of the sports game corresponds to a replay scene in order to identify each of the above situations. However, it is not limited thereto.

[0060] Again, referring to FIG. 2, the processor (202) can identify whether the designated visual object superimposed on at least some of the scenes corresponding to the advertisement content provided during the break time of the sports game is included in each of the plurality of partial video sequences in order to identify each of the above situations. For example, referring to FIG. 4c, the processor (202) can identify whether the designated visual object, such as the designated visual object (460), the designated visual object (465), the designated visual object (470), or the designated visual object (475), is included in at least some of the scenes corresponding to the advertisement content in order to identify each of the above situations. However, it is not limited thereto.

[0061] Referring again to FIG. 2, the processor (202) can train the neural network by providing the neural network with information regarding the designated visual object described through FIG. 3a to 4c. In one embodiment, the information regarding the designated visual object may include data regarding the shape of the designated visual object and data regarding the location where the designated visual object is provided. The location where the designated visual object is provided may be referred to as a designated location in that it means a location where the designated visual object is frequently provided.

[0062] For example, referring to FIG. 5a, in operation 502, the processor (202) can construct training data by labeling the designated visual object and the designated location. For example, the processor (202) can perform labeling based on user input to construct a region of interest (ROI) at a location where information (310) (or the designated visual object (310)) is displayed within the image (300) illustrated in FIG. 3a. Referring again to FIG. 5a, the designated location can be labeled in various formats. For example, the designated location can be labeled by specifying the area occupied by the designated visual object with coordinates to indicate the location of the area. For example, referring to FIG. 5b, the designated location can be labeled by specifying data for the coordinates of each of the corners (515-1), corner (515-2), corner (515-3), and corner (515-4) of the designated visual object (515) within the image (510) through user input. For another example, referring to FIG. 5b, the designated location can be labeled by specifying data for the coordinates of each of the corners (525-1), corner (525-2), corner (525-3), and corner (525-4) of the designated visual object (525) within the image (520) through user input. Referring again to FIG. 5a, the designated location can be labeled by indicating the area occupied by the designated visual object using at least one identifier among the identifiers for indicating each of the partial regions constituting the image including the designated visual object. For example, referring to FIG. 5c, the image (510) may be composed of a first partial region (530-1) to a ninth partial region (530-9).The above-mentioned designated location can be labeled by indicating that the first part area (530-1) indicated by the first identifier includes a designated visual object (515), among a first identifier for indicating a first part area (530-1), a second identifier for indicating a second part area (530-2), a third identifier for indicating a third part area (530-3), a fourth identifier for indicating a fourth part area (530-4), a fifth identifier for indicating a fifth part area (530-5), a sixth identifier for indicating a sixth part area (530-6), a seventh identifier for indicating a seventh part area (530-7), an eighth identifier for indicating an eighth part area (530-8), and a ninth identifier for indicating a ninth part area (530-9). For another example, referring to FIG. 5c, the image (520) may be composed of a first partial region (540-1) to a ninth partial region (540-9). The above-mentioned designated location may be labeled by indicating that the second part area (540-2) and the third part area (540-3), indicated by the second identifier and the third identifier among the first identifier for indicating the first part area (540-1), the second identifier for indicating the second part area (540-2), the third identifier for indicating the third part area (540-3), the fourth identifier for indicating the fourth part area (540-4), the fifth identifier for indicating the fifth part area (540-5), the sixth identifier for indicating the sixth part area (540-6), the seventh identifier for indicating the seventh part area (540-7), the eighth identifier for indicating the eighth part area (540-8), and the ninth identifier for indicating the ninth part area (540-9), include a designated visual object (525). However, it is not limited thereto.

[0063] Meanwhile, referring again to FIG. 5a, the processor (202) can configure training data based on the labeling.

[0064] In operation 560, the processor (202) may provide the configured training data to the neural network. In one embodiment, the neural network may perform training based on the training data. The neural network may perform the training by recognizing the shape of the designated visual object and the location where the designated visual object is displayed based on the training data.

[0065] As described above, the electronic device (102) may construct training data by labeling the designated visual object and the designated location to specify, using the neural network, a situation that provides at least a portion of the video sequence classified based on scenes, and may train the neural network based on the constructed training data. For example, the electronic device (102) may label coordinates constituting an area occupied by the designated visual object so that the neural network can specify the designated location as an ROI. As another example, the electronic device (102) may label at least one identifier to indicate each of at least one partial area within an image occupied by the designated visual object so that the neural network can specify the designated location as an ROI.

[0067] Referring again to FIG. 2, in one embodiment, the processor (202) may identify a portion of a video sequence indicating a designated scene and, before determining the type of the identified portion of the video sequence, identify whether the designated visual object is included within the designated location. For example, referring to FIG. 6, in operation 602, the processor (202) may identify a portion of a video sequence indicating a designated scene. For example, if the video sequence relates to a baseball game, the processor (202) may identify a portion of the video sequence containing the designated scene, which includes the actions of pitcher A within the baseball game among the scenes of the baseball game included in the video sequence. For example, the processor (202) may identify a portion of the video sequence containing the designated scene based on data obtained by executing a computer vision process for the video sequence using the neural network and statistical data of the baseball game. However, it is not limited thereto.

[0068] In operation 604, the processor (202) may determine the type of the identified part of the video sequence as a first type corresponding to the designated scene among the plurality of types, based on identifying, using the neural network, that the designated visual object is contained within the designated location after identifying the part of the video sequence and before determining the type of the part of the video sequence. In one embodiment, instead of identifying that the designated visual object is contained within the designated location, the processor (202) may determine the type of the identified part of the video sequence as a first type corresponding to the designated scene among the plurality of types, based on identifying that the designated visual object contained within the designated location does not contain the designated visual element (e.g., visual element (450-1) illustrated in FIG. 4b) after identifying the part of the video sequence and before determining the type of the part of the video sequence. Meanwhile, in one embodiment, the processor (202) can identify that the specified visual object is included within the specified location by using the neural network to perform image analysis only on the ROI corresponding to the specified location within each of the plurality of images constituting part of the video sequence.

[0069] In operation 606, the processor (202) may determine the type of the identified video sequence as a second type among the plurality of types, distinguished from the first type, based on identifying, using the neural network, that the designated visual object is not included within the designated location after identifying a part of the video sequence and before determining the type of the part of the video sequence. For example, the second type may be a type defined to classify content corresponding to a replay scene. For example, the second type may be a type defined to prevent duplicate provision with content classified as the first type. For example, the second type may be a type defined to provide content in a different space separated from the space where content classified as the first type is provided. However, it is not limited thereto. In one embodiment, the processor (202) may determine the type of the identified part of the video sequence as the second type among the plurality of types, based on identifying that the specified visual object included within the specified location includes the specified visual element (e.g., visual element (450-1) illustrated in FIG. 4b) after identifying the part of the video sequence and before determining the type of the part of the video sequence. Meanwhile, in one embodiment, the processor (202) may identify that the specified visual object is not included within the specified location by using the neural network to perform image analysis only on the ROI corresponding to the specified location within each of the plurality of images constituting the part of the video sequence.

[0070] As described above, the electronic device (102) can prevent duplicate provision of the same content by identifying whether the specified visual object is included within the specified location in each of the plurality of images constituting part of a video sequence containing a specified scene, and determining the type of part of the video sequence based on the result of the identification.

[0072] Referring again to FIG. 2, the processor (202) can identify a portion of a video sequence indicating a designated scene, determine the type of the identified portion of the video sequence based on the designated scene, and then identify whether the designated visual object is included within the designated location. For example, referring to FIG. 7, in operation 702, the processor (202) can identify a portion of a video sequence indicating the designated scene. For example, operation 702 may correspond to operation 602 illustrated in FIG. 6.

[0073] In operation 704, the processor (202) may determine, in response to identifying a part of the video sequence, that the type of the part of the video sequence is a first type corresponding to the specified scene.

[0074] In operation 706, the processor (202) may identify whether the designated visual object is included within the designated location in each of the plurality of images constituting the part of the video sequence, based on determining the part of the video sequence as the first type. In one embodiment, the processor (202) may identify whether the designated visual object is included within the designated location based on determining the type of the part of the video sequence as the first type in order to identify whether the content included within the part of the video sequence provided based on the first type causes discomfort. In one embodiment, instead of identifying whether the designated visual object is included within the designated location, the processor (202) may identify whether the designated visual object within the designated location includes the designated visual element. Meanwhile, the processor (202) may execute operation 708 based on identifying that the designated visual object is included within the designated location, and otherwise execute operation 710.

[0075] In operation 708, the processor (202) may maintain the type of part of the identified video sequence as the first type if the specified visual object is contained within the specified location, or if the specified visual object within the specified location does not contain the specified visual element.

[0076] In operation 710, the processor (202) can change the type of part of the identified video sequence to a second type distinct from the first type if the specified visual object is not included within the specified location, or if the specified visual object within the specified location includes the specified visual element.

[0077] As described above, the electronic device (102) can prevent inconvenience caused by the provision of the part of the video sequence by determining the type of part of the video sequence including a designated scene and then performing post-processing on the part of the video sequence.

[0079] A computer-readable storage medium according to one embodiment as described above may store one or more programs including instructions that cause the electronic device to receive information about a video sequence through the communication circuit from an external electronic device when executed by the processor (e.g., processor (202)) of the electronic device (e.g., electronic device (102)), identify a part of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence using a trained neural network, and determine the type of the part of the video sequence as a first type corresponding to the designated scene among a plurality of types or a second type distinguished from the first type among a plurality of types, depending on whether the designated visual object is included within a designated location within each of the plurality of images constituting the part of the video sequence using the trained neural network.

[0080] In one embodiment, the neural network may be trained based on data regarding the designated location and data regarding the shape of the designated visual object. For example, the data regarding the designated location may consist of coordinates for specifying the location of the designated visual object. As another example, the data regarding the designated location may consist of at least one identifier for indicating at least one grid among a plurality of grids dividing each of the plurality of images, in which the designated visual object may be included.

[0081] In one embodiment, the one or more programs may include instructions that cause the electronic device to, when executed by the at least one processor of the electronic device, to identify a part of the video sequence, determine the type of the part of the video sequence as the first type in response to identifying the part of the video sequence as the first type, and after determining the type of the part of the video sequence as the first type, identify whether the designated visual object is included within the designated location in each of the plurality of images, maintain the type of the part of the video sequence as the first type based on identifying that the designated visual object is included within the designated location, and change the type of the part of the video sequence from the first type to the second type based on identifying that the designated visual object is not included within the designated location.

[0082] In one embodiment, the one or more programs may include instructions that cause the electronic device to identify whether a designated visual object is included within a designated location by performing image analysis only on a region of interest (ROI) corresponding to a designated location within each of the plurality of images using the trained neural network when executed by the at least one processor of the electronic device.

[0083] In one embodiment, the video sequence may be a first video sequence, and the one or more programs may include instructions that further cause the electronic device to obtain a second video sequence distinct from the first video sequence based on a portion of the video sequence determined to be of the first type when executed by the at least one processor of the electronic device, and to obtain a third video sequence distinct from the first video sequence and the second video sequence based on a portion of the video sequence determined to be of the second type.

[0084] In one embodiment, the one or more programs may include instructions that further cause the electronic device to, when executed by the at least one processor of the electronic device, use the trained neural network to identify whether another visual object distinct from the designated visual object is included in another designated location within each of the plurality of images constituting part of the video sequence, and to determine the type of part of the video sequence as the first type based on identifying whether the designated visual object is included in the designated location and the other designated visual object is not included in the other designated location, and to determine the type of part of the video sequence as the second type based on identifying whether the designated visual object is not included in the designated location and the other designated visual object is not included in the other designated location, and to determine the type of part of the video sequence as a third type distinct from the first type and the second type based on identifying whether the designated visual object is not included in the designated location and the other designated visual object is included in the other designated location, and the designated visual object is, if the video sequence contains content related to a sports game, the sports It may be used to indicate the status of the sports game during a live broadcast of the game, and the other designated visual object may be used to indicate the status of the sports game on advertising content provided during the break time of the sports game if the video sequence includes content related to the sports game.

[0085] In one embodiment, the designated visual object may include information superimposed on a screen for a live broadcast to indicate the state of the sports game during the live broadcast of the sports game, or information superimposed on a screen for an advertisement to indicate the state of the sports game during an advertisement provided between a part of the live broadcast and another part of the live broadcast, where the video sequence includes content related to the sports game.

[0086] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine the type of a portion of the video sequence as the second type among the plurality of types based on identifying that the specified visual element is included within the specified visual object using the trained neural network when executed by the at least one processor of the electronic device, and to determine the type of a portion of the video sequence as the first type among the plurality of types based on identifying that the specified visual element is not included within the specified visual object using the trained neural network.

[0088] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0089] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.

[0090] One or more of these programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), magnetic disc storage device, optical storage device (e.g., Compact Disc-ROM (CD-ROM), Digital Versatile Discs (DVDs)), magnetic cassette, or a combination thereof. Or, they may be stored in memory composed of some or all of these. Additionally, the above program may be stored on an attachable storage device accessible via a communication network such as the Internet, Intranet, Local Area Network (LAN), Wide LAN (WLAN), or Storage Area Network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0091] In the specific embodiments of the present disclosure described above, the components of the electronic device included in the present disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression of said components is selected to suit the circumstances presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0092] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

Claims

Claim 1 In a computer-readable storage medium storing one or more programs, the one or more programs, when executed by at least one processor of an electronic device having a communication circuit, receive information about a video sequence from an external electronic device through the communication circuit, use a trained neural network to identify a portion of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence, use the trained neural network to identify whether a designated visual object is included within a designated location within each of a plurality of images constituting the portion of the video sequence, and based on identifying that the designated visual object is included within the designated location, determine the type of the portion of the video sequence as a first type corresponding to the designated scene among a plurality of types, and based on identifying that the designated visual object is not included within the designated location, identify whether another visual object distinct from the visual object is included within another designated location within each of a plurality of images constituting the portion of the video sequence, and based on identifying that the other designated visual object is included within the other designated location, determine the type of the portion of the video sequence as a second type distinct from the first type, and A computer-readable storage medium comprising instructions that cause the electronic device to determine a type of part of the video sequence as a third type distinct from the first type and the second type, based on identifying that the other designated visual object is not contained within another designated location. Claim 2 A computer-readable storage medium according to claim 1, wherein the neural network is trained based on data for the designated location, data for the shape of the designated visual object, data for the other designated location, and the shape of the other designated visual object. Claim 3 A computer-readable storage medium according to claim 2, wherein data for the designated location is composed of coordinates for specifying the location of the designated visual object, and data for the other designated location is composed of coordinates for specifying the location of the other designated visual object. Claim 4 A computer-readable storage medium according to claim 2, wherein data for the designated location is composed of at least one identifier for indicating at least one grid among a plurality of grids dividing each of the plurality of images in which the designated visual object may be included, and data for the other designated location is composed of at least one other identifier for indicating at least one other grid among the plurality of grids in which the other designated visual object may be included. Claim 5 In claim 1, the one or more programs, when executed by the at least one processor of the electronic device, in response to identifying a portion of the video sequence, determine the type of the portion of the video sequence as the first type; after determining the type of the portion of the video sequence as the first type, identify whether the designated visual object is included within the designated location in each of the plurality of images; based on identifying that the designated visual object is included within the designated location, maintain the type of the portion of the video sequence as the first type; based on identifying that the designated visual object is not included within the designated location, identify whether the other visual object is included within the other designated location in each of the plurality of images constituting the portion of the video sequence; based on identifying that the other designated visual object is included within the other designated location, change the type of the portion of the video sequence from the first type to the second type; and based on identifying that the other designated visual object is not included within the other designated location, change the type of the portion of the video sequence from the first type to the third type. A computer-readable storage medium comprising instructions that cause the above electronic device. Claim 6 A computer-readable storage medium according to claim 1, wherein the one or more programs, when executed by the at least one processor of the electronic device, identify whether a designated visual object is included within a designated location by performing image analysis only on a region of interest (ROI) corresponding to a designated location within each of the plurality of images using the trained neural network, and identify whether a different designated visual object is included within a different designated location by performing image analysis only on a different ROI corresponding to a different designated location within each of the plurality of images using the trained neural network. Claim 7 A computer-readable storage medium according to claim 1, wherein the video sequence is a first video sequence, and the one or more programs further include instructions that cause the electronic device to obtain a second video sequence distinguished from the first video sequence based on a portion of the video sequence determined to be of the first type when executed by the at least one processor of the electronic device, and to obtain a third video sequence distinguished from the first video sequence and the second video sequence based on a portion of the video sequence determined to be of the second type. Claim 8 A computer-readable storage medium according to claim 1, wherein the designated visual object is used to indicate the state of the sports game during a live broadcast of the sports game when the video sequence includes content related to the sports game, and the other designated visual object is used to indicate the state of the sports game on advertising content provided during a break time of the sports game when the video sequence includes content related to the sports game. Claim 9 A computer-readable storage medium according to claim 1, wherein the designated visual object comprises information superimposed on a screen for a live broadcast to indicate the state of the sports game during a live broadcast of the sports game, or information superimposed on a screen for an advertisement broadcast to indicate the state of the sports game during an advertisement broadcast provided between a part of the live broadcast and another part of the live broadcast. Claim 10 A computer-readable storage medium according to claim 1, wherein the one or more programs further include instructions that cause the electronic device to determine the type of a portion of the video sequence as the second type among the plurality of types based on identifying that the specified visual element is included within the specified visual object using the trained neural network when executed by the at least one processor of the electronic device, and to determine the type of a portion of the video sequence as the first type among the plurality of types based on identifying that the specified visual element is not included within the specified visual object using the trained neural network. Claim 11 A method executed within an electronic device having a communication circuit, comprising: receiving information about a video sequence from an external electronic device through the communication circuit; using a trained neural network to identify a portion of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence; using the trained neural network to identify whether the designated visual object is included within a designated location within each of a plurality of images constituting the portion of the video sequence; determining the type of the portion of the video sequence as a first type among a plurality of types corresponding to the designated scene based on identifying that the designated visual object is included within the designated location; identifying whether another visual object distinct from the visual object is included within another designated location within each of a plurality of images constituting the portion of the video sequence based on identifying that the designated visual object is not included within the designated location; determining the type of the portion of the video sequence as a second type distinct from the first type based on identifying that the other designated visual object is included within the other designated location; and the other designated visual object within the other designated location A method comprising the operation of determining a type of part of the video sequence as a third type distinct from the first type and the second type, based on identifying what is not included. Claim 12 A method according to claim 11, wherein the neural network is trained based on data for the designated location, data for the shape of the designated visual object, data for the other designated location, and the shape of the other designated visual object. Claim 13 A method according to claim 12, wherein data for the designated location is composed of coordinates for specifying the location of the designated visual object, and data for the other designated location is composed of coordinates for specifying the location of the other designated visual object. Claim 14 A method according to claim 12, wherein data for the designated location is composed of at least one identifier for indicating each of at least one grid among a plurality of grids dividing each of the plurality of images, wherein the designated visual object may be included therein, and data for the other designated location is composed of at least one other identifier for indicating each of at least one other grid among the plurality of grids, wherein the other designated visual object may be included therein. Claim 15 A method according to claim 11, further comprising: an action of determining the type of the part of the video sequence as the first type in response to identifying the part of the video sequence; an action of determining whether the designated visual object is included within the designated location within each of the plurality of images after determining the type of the part of the video sequence as the first type; an action of maintaining the type of the part of the video sequence as the first type based on identifying that the designated visual object is included within the designated location; an action of determining whether the other visual object is included within the other designated location within each of the plurality of images constituting the part of the video sequence based on identifying that the designated visual object is not included within the designated location; an action of changing the type of the part of the video sequence from the first type to the second type based on identifying that the other designated visual object is included within the other designated location; and an action of changing the type of the part of the video sequence from the first type to the third type based on identifying that the other designated visual object is not included within the other designated location. Claim 16 A method according to claim 11, further comprising: an operation of identifying whether a designated visual object is included within a designated location by performing image analysis only on a region of interest (ROI) corresponding to a designated location within each of the plurality of images using the trained neural network; and an operation of identifying whether a different designated visual object is included within a different designated location by performing image analysis only on a different ROI corresponding to a different designated location within each of the plurality of images using the trained neural network. Claim 17 The method of claim 11, wherein the video sequence is a first video sequence, and the method further comprises the operation of obtaining a second video sequence distinguished from the first video sequence based on a part of the video sequence determined to be of the first type, and the operation of obtaining a third video sequence distinguished from the first video sequence and the second video sequence based on a part of the video sequence determined to be of the second type. Claim 18 A method according to claim 11, wherein the designated visual object is used to indicate the state of the sports game during a live broadcast of the sports game when the video sequence includes content related to the sports game, and the other designated visual object is used to indicate the state of the sports game on advertising content provided during a break time of the sports game when the video sequence includes content related to the sports game. Claim 19 A method according to claim 11 further comprising the operation of determining the type of a part of the video sequence as the second type among the plurality of types based on identifying that the specified visual element is included within the specified visual object using the trained neural network, and the operation of determining the type of a part of the video sequence as the first type among the plurality of types based on identifying that the specified visual element is not included within the specified visual object using the trained neural network. Claim 20 In an electronic device, a communication circuit; a memory configured to store instructions; The system includes a processor, wherein, when executing the instructions, the processor receives information about a video sequence from an external electronic device via the communication circuit, and using a trained neural network, identifies a portion of the video sequence indicating a designated scene among a plurality of scenes indicated by the video sequence, and using the trained neural network, identifies whether the designated visual object is included within a designated location within each of the plurality of images constituting the portion of the video sequence, and based on identifying that the designated visual object is included within the designated location, determines the type of the portion of the video sequence as a first type among a plurality of types corresponding to the designated scene, and based on identifying that the designated visual object is not included within the designated location, identifies whether another visual object distinct from the visual object is included within another designated location within each of the plurality of images constituting the portion of the video sequence, and based on identifying that the other designated visual object is included within the other designated location, determines the type of the portion of the video sequence as a second type distinct from the first type, and the other designated visual object is not included within the other designated location An electronic device configured to determine, based on identifying, a type of part of the video sequence as a third type distinct from the first type and the second type.

Images