Data analysis method and electronic device for performing same method

Abstract

A data analysis method and an electronic device for performing the method are disclosed. An electronic device according to various embodiments may comprise: at least one processor; and a memory which is electrically connected to the at least one processor and stores at least one instruction executed by the processor, wherein the at least one processor, when the at least one instruction is executed, causes the electronic device to: receive, from a plurality of cameras, a plurality of images obtained by capturing images of an image capturing space; for each configured frame, identify at least one object included in the plurality of images; for each of the configured frames, determine a position of the at least one object in a three-dimensional coordinate system configured with respect to the image capturing space; determine data related to the position of the at least one object; and transmit the data to a user terminal.

Description

Data analysis method and electronic device performing the method

[0001] The disclosure below relates to a data analysis method and an electronic device for performing the method.

[0002]

[0003] As public interest in sports events grows, the demand for analytics data related to sports events is also increasing.

[0004] Analysis data can be provided to the general public, as well as used for player training and management. Furthermore, when broadcasting sports events in real time, analyzed data can be provided during the broadcast.

[0005] For data analysis, for example, a soccer player could wear a separate device while playing, and information about the player's movements and actions could be collected using the device. For data analysis, information about the player's movements and actions could be collected using a variety of methods.

[0006] The background technology described above is technology that the inventor possessed or acquired in the process of deriving the disclosure of the present application, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the present application.

[0007]

[0008] According to various embodiments, a data analysis method capable of analyzing data from a captured image and providing the analyzed data without a separate device and an electronic device performing the method can be provided.

[0009] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist.

[0010]

[0011] An electronic device according to various embodiments includes at least one processor and a memory electrically connected to the at least one processor and storing at least one command executed by the processor, wherein the at least one processor, when the at least one command is executed, causes the electronic device to receive a plurality of images captured with respect to a shooting space from a plurality of cameras, identify at least one object included in the plurality of images for each set frame, determine a position of the at least one object in a three-dimensional coordinate system set with respect to the shooting space for each set frame, determine data related to the position of the at least one object, and transmit the data to a user terminal.

[0012] The at least one processor can determine two-dimensional coordinates of the at least one object using each of the plurality of images, and can determine a position of the at least one object using the two-dimensional coordinates of the at least one object.

[0013] The at least one processor can recognize additional information using the plurality of images, and determine the data based on the location of the at least one object and the additional information.

[0014] The at least one processor may determine prediction data regarding the at least one object using at least one of the location of the at least one object and the data.

[0015] The at least one processor can identify a relay image regarding the shooting space and the at least one object, and transmit the relay image and the data to the user terminal.

[0016] The at least one processor may receive a plurality of second images for generating the relay image from the plurality of cameras, and generate the relay image as at least one image among the plurality of second images based on the type and location of the at least one object.

[0017] The at least one processor can control at least one camera among the plurality of cameras based on the type and location of the at least one object.

[0018] The at least one processor can generate a relay image in which the data is visually overlapped with the relay image, and transmit the overlapped relay image to the user terminal.

[0019] The at least one processor may transmit the data to the user terminal in response to a request received from the user terminal.

[0020] A data analysis method according to various embodiments may include an operation of receiving a plurality of images captured from a plurality of cameras with respect to a shooting space, an operation of identifying at least one object included in the plurality of images for each set frame, an operation of determining a position of the at least one object in a three-dimensional coordinate system set with respect to the shooting space for each set frame, an operation of determining data related to the position of the at least one object, and an operation of transmitting the data to a user terminal.

[0021] The operation of determining the position of the at least one object may include an operation of determining two-dimensional coordinates of the at least one object using each of the plurality of images and an operation of determining the position of the at least one object using the two-dimensional coordinates of the at least one object.

[0022] The above data analysis method further includes an operation of recognizing additional information using the plurality of images, and the operation of determining the data can determine the data based on the location of the at least one object and the additional information.

[0023] The above data analysis method may further include an operation of determining prediction data regarding the at least one object by using at least one of the location of the at least one object and the data.

[0024] The above data analysis method may further include an operation of identifying a relay image regarding the shooting space and the at least one object, and the operation of transmitting to the user terminal may include an operation of transmitting the relay image and the data to the user terminal.

[0025] The operation of receiving the relay image may include an operation of receiving a plurality of second images for generating the relay image from the plurality of cameras, and an operation of generating the relay image using at least one image from the plurality of second images based on the type and location of the at least one object.

[0026] The above data analysis method may further include an operation of controlling at least one camera among the plurality of cameras based on the type and location of the at least one object.

[0027] The operation of transmitting the above relay image and the above data to the user terminal may include an operation of generating a relay image in which the data is visually overlapped with the relay image and an operation of transmitting the overlapped relay image to the user terminal.

[0028] The operation of transmitting to the user terminal may include an operation of transmitting the data to the user terminal in response to a request received from the user terminal.

[0029]

[0030] According to various embodiments, a data analysis method and an electronic device performing the method can collect information about the position and movement of an object from a captured image, and analyze the collected information to determine data.

[0031] According to various embodiments, the data analysis method and the electronic device performing the method may provide analyzed data or provide analyzed data in combination with a relay image.

[0032]

[0033] FIG. 1A is a schematic block diagram of an electronic device, a user terminal, and a camera module according to various embodiments.

[0034] FIG. 1b is a schematic block diagram of an electronic device according to various embodiments.

[0035] FIG. 2 is a flowchart of a data analysis method performed by an electronic device according to various embodiments.

[0036] FIG. 3 is a drawing showing a plurality of detection cameras and a plurality of shooting cameras arranged in a shooting space according to various embodiments.

[0037] FIGS. 4A, 4B, 4C, and 4D are diagrams illustrating an operation of an electronic device recognizing an object according to various embodiments.

[0038] FIGS. 5A, 5B, 5C, and 5D are diagrams illustrating an operation of an electronic device according to various embodiments to determine the position of an object.

[0039] FIG. 6 is a diagram illustrating an operation of an electronic device generating data according to various embodiments.

[0040] FIGS. 7A, 7B, 7C, and 7D are diagrams illustrating data provided by an electronic device according to various embodiments.

[0041]

[0042] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0043] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0044] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0045] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this document, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can each include any one of the items listed together in that phrase, or all possible combinations thereof. In this specification, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0046] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0047] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or portion of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0048] The term "~ component" as used in this document refers to a software or hardware component such as an FPGA or ASIC, and the "~ component" performs certain roles. However, the "~ component" is not limited to software or hardware. The "~ component" may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. For example, the "~ component" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and "~ components" may be combined into a smaller number of components and "~ components" or further separated into additional components and "~ components." Furthermore, the components and "~ components" may be implemented to execute one or more CPUs within a device or a secure multimedia card. Additionally, '~bu' may include one or more processors.

[0049] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0050]

[0051] FIG. 1A is a schematic block diagram of an electronic device (100), a user terminal (200), and a camera (300) according to various embodiments.

[0052] As shown in FIG. 1a, the system (10) may include an electronic device (100), a user terminal (200), and a camera (300).

[0053] Referring to FIG. 1A, an electronic device (100) according to various embodiments may receive multiple images from a camera (300). For example, the camera (300) may include multiple detection cameras and multiple photographing cameras.

[0054] For example, the electronic device (100) may receive multiple images from multiple detection cameras to identify the type and location of an object. The electronic device (100) may receive multiple second images from multiple shooting cameras to generate a relay image.

[0055] The electronic device (100) can identify at least one object existing in a shooting space using multiple images. For example, the electronic device (100) can input multiple images into a trained model (140) to identify at least one object. For example, when multiple images are input, the model (140) can be trained to output the types of objects included in the multiple images (e.g., players, balls, referees, goal posts, table tennis tables, etc.). The electronic device (100) can identify an object for each set frame.

[0056] For example, the electronic device (100) can determine the location of an identified object using multiple images. For example, the electronic device (100) can determine the location of the identified object in a three-dimensional coordinate system set with respect to the shooting space. The electronic device (100) can determine the location of the object for each set frame.

[0057] For example, the electronic device (100) can determine the two-dimensional coordinates of at least one object using each of a plurality of images. For example, the electronic device (100) can determine a pixel (or pixel area) where the object is located within the image as the two-dimensional coordinates of the object.

[0058] For example, the electronic device (100) can determine the location of at least one object using the two-dimensional coordinates of at least one object. The electronic device (100) can determine the location (e.g., three-dimensional coordinates) of the object using the locations of a plurality of detection cameras and the two-dimensional coordinates of the object (in a plurality of images) within a set three-dimensional coordinate system.

[0059] For example, the electronic device (100) can determine data related to the location of at least one object. For example, the electronic device (100) can generate data using the type and location of the object determined for each set frame.

[0060] For example, the electronic device (100) can receive a relay image from an external electronic device. The electronic device (100) can transmit the relay image and data to the user terminal (200). For example, the electronic device (100) can transmit a relay image with visualized data overlapping the relay image to the user terminal (200). For example, the electronic device (100) can transmit a relay image combined with visualized data to the user terminal (200).

[0061] For example, the electronic device (100) can transmit the determined relay image and data to the user terminal (200). The electronic device (100) can determine a control signal based on the type and location of the identified object obtained using the camera (300). For example, the electronic device (100) can input the location of the identified object into a learned second model to determine the control signal. The second model can be trained to output a control signal when the type and location of the identified object are input.

[0062] For example, the electronic device (100) can transmit a control signal to a plurality of photographing cameras. The electronic device (100) can control the plurality of photographing cameras using the control signal.

[0063] For example, the electronic device (100) may determine a relay image based on a control signal. The electronic device (100) may determine a relay image using at least one second image among a plurality of second images received from a plurality of shooting cameras.

[0064] For example, the control signal may include information about a camera selected from among a plurality of cameras for shooting to generate a relay image, information about the operation of the selected camera for shooting (e.g., PTZ (pan, tilt, zoom) values, direction / angle / magnification values, etc.). The electronic device (100) may determine a camera for shooting to obtain a second image to be used as a relay image according to the control signal. The electronic device (100) may determine a second image obtained from a camera selected from among a plurality of cameras for shooting to be a relay image.

[0065] For example, a plurality of shooting cameras may be controlled according to a control signal received from an electronic device (100). For example, the plurality of shooting cameras may each operate according to information included in the control signal (e.g., PTZ (pan, tilt, zoom) values, direction / angle / magnification values, etc.). For example, the plurality of shooting cameras may each include an operation module (e.g., an actuator, a motor, etc.). Each of the operation modules of the plurality of shooting cameras may operate according to the control signal.

[0066] For example, the electronic device (100) can transmit a relay image to a user terminal (200). The electronic device (100) can receive information about the user terminal (200) from the user terminal (200) (e.g., terminal type, terminal specifications, user information, etc.). The electronic device (100) can post-process the relay image based on the information about the user terminal (200). The electronic device (100) can transmit the post-processed relay image to the user terminal (200).

[0067] For example, post-processing may include changing the format of an image, compressing an image, or resizing an image. For example, post-processing may include changing the content of a broadcast image, such as adding additional images (e.g., advertisements) or auxiliary objects to the broadcast image.

[0068]

[0069] FIG. 1b is a schematic block diagram of an electronic device (100) according to various embodiments.

[0070] Referring to FIG. 1B, an electronic device (100) according to various embodiments may include a processor (110), a memory (120), a communication circuit (130), and a model (140).

[0071] The memory (120) can store various data used by at least one component (e.g., processor (110)) of the electronic device (100). For example, the data can include input data or output data for software (or programs, applications, etc.) and commands related thereto. The memory (120) can include volatile memory or non-volatile memory.

[0072] For example, the processor (110) may execute software (or a program, an application, etc.) to control at least one other component of the electronic device (100) connected to the processor (110). The processor (110) may execute the software and display or output the processed result through a display module or an audio output module (e.g., a speaker). The processor (110) may execute the software to perform data processing or calculation. For example, the processor (110) may store commands or data received from other components of the electronic device (100) (e.g., the communication circuit (130), the first model (140), the second model) in the memory (120). The processor (110) may process commands or data stored in the memory (120) and store result data in the memory (110).

[0073] For example, the communication circuit (130) can establish a direct communication channel or a wireless communication channel between the electronic device (100) and an external electronic device (e.g., user terminal (200), camera (300)), and support wired / wireless communication through the established communication channel.

[0074] For example, the electronic device (100) can establish a communication channel with the user terminal (200) using the communication circuit (130) and transmit data (or data combined with a relay image). For example, the electronic device can establish a communication channel with the camera (300) using the communication circuit (130) and receive a plurality of first images (and / or a plurality of second images) from the camera (300).

[0075] For example, the model (140) can be trained to output the types of objects included in a plurality of images (e.g., a plurality of first images) when input. For example, the description of a neural network model for recognizing objects in images known to the model (140) can be substantially identically applied. For example, the description of a known object detection model or object recognition model can be substantially identically applied to the model (140).

[0076] According to one embodiment, the electronic device (100) can determine the location of at least one identified object. For example, the electronic device (100) can determine the location of the object identified by the first model (140). The electronic device (100) can determine the location (e.g., 3D coordinates) of the identified object within a set 3D coordinate system.

[0077] For example, the electronic device (100) can input multiple first images into the model (140) to identify additional information. For example, the additional information may include player numbers, referee gestures, and information displayed on the scoreboard. The additional information is not limited to the examples above and may include objects and / or information that can be recognized by analyzing the multiple first images.

[0078] For example, the electronic device (100) may determine data related to the location of at least one object. For example, the data may include information related to the location of the object. The data may include, but is not limited to, the object's average speed, travel distance, maximum speed, and movement trajectory.

[0079] For example, the electronic device (100) can determine data using the location and / or additional information of at least one object. For example, the electronic device (100) can determine the score status of each team / player using additional information (e.g., information displayed on an electronic scoreboard). For example, the electronic device can determine the movement trajectory of a player or ball using additional information (e.g., jersey number) and the location of an object (player).

[0080] In one embodiment, the information contained in the data may vary depending on the type of event (e.g., a sporting event, a ceremony, etc.) currently taking place at the filming location. For example, the data may vary depending on the type of sporting event being played (e.g., a soccer game, a baseball game, a table tennis game, a track and field event).

[0081] If a soccer match is in progress at the filming location, the data may include information such as each player's average speed, top speed, distance traveled, movement trajectory, number of fouls, number of goals scored, pass completion percentage, number / percentage of shots on target, number of shots, and each team's attack route percentage.

[0082] If a table tennis match is in progress at the filming location, the data may include information such as the number / rate of serve points for each player, attack pattern rate, attack area rate, average speed of the ball, maximum speed of the ball, and information about the trajectory of the table tennis ball.

[0083] The specific information contained in the data is not limited to the examples above. The specific information contained in the data may be determined based on settings, and the electronic device (100) may determine the data using the object's location and / or additional information.

[0084] For example, the electronic device (100) may determine predicted data using at least one of the object's location, additional information, external information, and data, or a combination thereof. For example, the predicted data may include predicted information about an event occurring at the filming location, such as the expected probability of winning a game. For example, the external information may represent information received by the electronic device (100) from an external source. For example, in the case of a sports game, the external information may include each player's winning percentage, physical information (height, weight, age, etc.), game patterns, etc.

[0085] For example, the electronic device (100) may output predicted data using a model (e.g., a predictive model) trained to output predicted data. For example, the predictive model may be trained to output predicted data when at least one of an object's location, additional information, external information, and data, or a combination thereof, is input. Various known artificial neural network models may be applied to the predictive model.

[0086] For example, the electronic device (100) may transmit data to the user terminal (200). For example, the electronic device (100) may receive a request from the user terminal (200) regarding information to be transmitted to the user terminal (200) among the information included in the data. In response to the received request, the electronic device (100) may transmit data including the requested information to the user terminal (200).

[0087] For example, the electronic device (100) can receive a relay image or generate a relay image. The electronic device (100) can transmit the relay image and data to the user terminal (200).

[0088] For example, the electronic device (100) can overlap visualized data with a relay image. The electronic device (100) can transmit the relay image with the visualized data overlapped to the user terminal (200).

[0089] For example, the electronic device (100) can combine visualized data with a relay image. The electronic device (100) can transmit the relay image combined with the visualized data to the user terminal (200).

[0090] For example, the second model can be trained to output a control signal for generating a relay image when the type and location of an object are input. For example, the type and location of an object can be input to the second model for each set frame. The second model can be trained to output a control signal using the type and location of the object input for each set frame.

[0091] For example, the electronic device (100) can determine the type of at least one object included in a plurality of first images for each set frame. The electronic device (100) can determine the location of at least one object included in a plurality of first images for each set frame. The electronic device (100) can input the type and location of at least one object for each set frame into a second model to determine a control signal.

[0092] For example, when a table tennis match is being played in a shooting space, the electronic device (100) can identify the type of each object (e.g., player, referee, table tennis ball, table tennis table, scoreboard, etc.) using a plurality of first images acquired from a plurality of detection cameras. The electronic device (100) can determine the location of each identified object.

[0093] For example, the control signal may include information for controlling the operation of a plurality of cameras for shooting, and information regarding a second image to be used for relay imaging among a plurality of second images.

[0094] For example, training data for training a second model may include input data and correct answer data. For example, the training data may include the types and locations of each object acquired in a shooting space equipped with multiple detection cameras and multiple recording cameras, as well as control signals for generating relay images.

[0095] For example, the learning data may include the types and locations of each object, as well as relay images, acquired from a shooting space where multiple detection cameras and multiple recording cameras are installed. The relay images may be understood as being substantially identical to the control signals for generating the relay images. For example, the relay images may indicate the recording camera that captured the image used in the relay image, and the operation information of the recording camera (e.g., PTZ values, direction / angle / magnification, etc.).

[0096] The second model can be trained to output a control signal for generating a relay signal using the type and location of each input object. The electronic device (100) can generate a relay image based on the location of each object using the control signal.

[0097]

[0098] FIG. 2 is a flowchart of a data analysis method performed by an electronic device (100) according to various embodiments.

[0099] Referring to FIG. 2, an electronic device (100) according to various embodiments may receive a plurality of first images captured with respect to a shooting space from a plurality of detection cameras in operation (210).

[0100] For example, multiple detection cameras can be positioned and fixed around the shooting space. The positions, directions, angles, and magnifications of the multiple detection cameras can be fixed.

[0101] For example, multiple cameras for capturing images may be positioned around the shooting space and their positions may be fixed. The direction, angle, and magnification of the multiple detection cameras may be controlled.

[0102] For example, the electronic device (100) can identify at least one object included in a plurality of first images for each frame set in operation (220). For example, the electronic device (100) can input a plurality of first images into a learned model (140) to identify at least one object. When a plurality of first images are input, the model (140) can be trained to identify the type of each object included in the plurality of first images.

[0103] For example, the electronic device (100) can determine the position of at least one object in a three-dimensional coordinate system set with respect to the shooting space for each frame set in the operation (230). For example, the electronic device (100) can determine the position (three-dimensional coordinate) of the object by using the two-dimensional (pixel) coordinates of each object in the plurality of first images and the three-dimensional coordinates (of the set three-dimensional coordinate system) of the plurality of detection cameras.

[0104] For example, the electronic device (100) may determine data related to the location of at least one object in operation (240). For example, the data may include information related to the object and / or the movement of the object. The electronic device (100) may determine the type and location of the object for each set frame. The electronic device (100) may determine data related to the object by using the type and location (or changes in the time-series location) of the object identified for each set frame.

[0105] For example, the electronic device (100) can transmit data to the user terminal (200) in operation (250).

[0106] The operations (210) to (250) illustrated in FIG. 2 can be substantially identically performed by the processor (110) of the electronic device (100).

[0107] The operations (210) to (250) illustrated in FIG. 2 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0108]

[0109] FIG. 3 is a drawing showing a plurality of detection cameras (300a) and a plurality of shooting cameras (300b) arranged in a shooting space according to various embodiments.

[0110] Figure 3 is a drawing showing a plurality of detection cameras (300a) and a plurality of shooting cameras (300b) deployed when the shooting location is a soccer stadium.

[0111] For example, the electronic device (100) can receive a plurality of first images from a plurality of detection cameras (300a) positioned at a shooting location. The electronic device (100) can identify at least one object using the plurality of first images. The electronic device (100) can determine the location of at least one identified object using the plurality of first images. The electronic device (100) can determine the location (e.g., 3D coordinates) of at least one object in a set 3D coordinate system.

[0112] For example, the electronic device (100) may input the type and location of at least one object into a learned model (e.g., a second model) to determine a control signal for generating a relay image. The electronic device (100) may receive a plurality of second images from a plurality of capturing cameras (300b). The electronic device (100) may generate a relay image using at least one second image among the plurality of second images according to the control signal.

[0113] The electronic device (100) can transmit the generated relay image to the user terminal (200).

[0114] The arrangement of the plurality of detection cameras (300a) and the plurality of photographing cameras (300b) illustrated in FIG. 3 is exemplary and is not limited to the example illustrated in FIG. 3.

[0115]

[0116] FIG. 4a, FIG. 4b, FIG. 4c, and FIG. 4d are drawings showing an operation of an electronic device (100) recognizing an object according to various embodiments.

[0117] FIG. 4a is a drawing showing a camera (300) (e.g., multiple detection cameras (300a), multiple shooting cameras (300b)), an object (410), and an environment (420-1, 420-2) placed at a shooting location.

[0118] For example, the electronic device (100) can identify an object (410) using a plurality of first images received from a plurality of detection cameras (300a). The electronic device (100) can determine the location of the object (410) using a plurality of first images received from a plurality of detection cameras (300a).

[0119] For example, the electronic device (100) can identify additional information using a plurality of first images. For example, the electronic device (100) can identify additional information from objects included in the plurality of first images. For example, the additional information may include player jersey numbers, referee gestures, and information displayed on an electronic scoreboard. The electronic device (100) can identify the environment (420-1, 420-2) from the plurality of first images.

[0120] In the above example, the electronic device (100) can input a plurality of first images into a learned model (e.g., model (140)) to identify the environment (420-1, 420-2) or additional information.

[0121] FIG. 4b is a drawing showing a plurality of first images (311, 313, 315, 317) received from a plurality of detection cameras (300a).

[0122] In FIG. 4b, each of the plurality of first images (311, 313, 315, 317) may include an object (411). Each of the plurality of first images (311, 313, 315, 317) may be captured by a plurality of detection cameras (300a) positioned at different locations. The locations of the objects (411) within each of the plurality of first images (311, 313, 315, 317) may be different from each other.

[0123] The electronic device (100) can identify an object (411) by inputting a plurality of first images (311, 313, 315, 317) into the first model (140). The electronic device (100) can determine the type of the object (411) as a player.

[0124] Fig. 4c is a diagram showing a plurality of first images (311, 313, 315, 317) received from a plurality of detection cameras (300a). Unlike Fig. 4b, Fig. 4c shows that the first images (311) and (313) include an object (411), but the first images (315) and (317) do not include the object (411). For example, if the detection cameras that captured the first images (315) and (317) are not facing the object (411), the first images (315) and (317) may not include the object (411). For example, even if the detection camera that captured the first image (315) and the first image (317) is directed toward the object (411), if the object (411) is obscured by an obstacle or another object, the first image (315) and the first image (317) may not include the object (411).

[0125] As shown in FIG. 4c, even if the first image (311) and the first image (313) include an object (411), but the first image (315) and the first image (317) do not include an object (411), the electronic device (100) can identify the object (411) using a plurality of first images (311, 313, 315, 317).

[0126] FIG. 4d is a drawing showing objects (411, 413, 415, 417) recognized using a plurality of first images (311, 313, 315, 317).

[0127] The electronic device (100) can identify objects (411, 413, 415, 417), as shown in FIG. 4D. For example, the electronic device (100) can identify objects (411) and (413) as players. For example, the electronic device (100) can identify object (415) as a referee. For example, the electronic device (100) can identify object (417) as a soccer ball.

[0128]

[0129] FIG. 5a, FIG. 5b, FIG. 5c, and FIG. 5d are drawings showing an operation of an electronic device (100) determining the position of an object according to various embodiments.

[0130] FIG. 5a is a diagram showing a three-dimensional coordinate system (500) and object coordinates (520, 530) according to various embodiments.

[0131] For example, a three-dimensional coordinate system (500) may be set with respect to the shooting space. For example, a plurality of detection cameras (300a) may be positioned around the shooting space. Through calibration, distortion of a plurality of first images acquired from a plurality of detection cameras (300a) positioned around the shooting space may be eliminated.

[0132] For example, the origin (510) of the three-dimensional coordinate system (500) can be determined as an arbitrary location in the shooting location. As shown in Fig. 5a, one corner of a soccer field can be determined as the origin (510) of the three-dimensional coordinate system (500).

[0133] For example, a calibration plate placed in a shooting location can be captured by at least one detection camera. In one or more images captured by the calibration plate, pixels indicating the calibration plate can be understood as capturing the same location. By repeatedly capturing calibration plates placed in different locations within the shooting location using one or more detection cameras, the positions of each detection camera can be estimated (or determined), and a three-dimensional coordinate system (with an arbitrary location within the shooting location as its origin) can be established.

[0134] For example, when the origin (510) of the 3D coordinate system (500) is determined, the 3D coordinate system (500) can be set with respect to the shooting location. The positions (e.g., 3D coordinates) of the plurality of detection cameras (300a) can be determined in the 3D coordinate system (500). The positions (e.g., 3D coordinates) of the plurality of shooting cameras (300b) can be determined in the 3D coordinate system (500).

[0135] As shown in FIG. 5A, the electronic device (100) can determine the position of an object (e.g., 3D coordinates). For example, the electronic device (100) can determine the position of a person as coordinates (520) ((x24, y11, z1) or (24, 11, 1)). The electronic device (100) can determine the position of a ball as coordinates (520) ((x64, y47, z3) or (64, 47, 3)).

[0136] FIG. 5b is a drawing showing the coordinates (321, 323, 325, 327) of an object in a plurality of first images (311, 313, 315, 317) received from a plurality of detection cameras (300a).

[0137] A plurality of first images (311, 313, 315, 317) are two-dimensional, and the coordinates of the object (321, 323, 325, 327) within each image can be expressed as two-dimensional coordinates (e.g., (x1, y1), (x2, y2), (x3, y3), (x4, y4)).

[0138] For example, the position of the detection camera that captured the first image (311) may be (x1, y1, z1). Using the coordinates (321) of the object in the first image (311), the direction and / or angle of viewing the object from the position (x1, y1, z1) of the detection camera may be calculated.

[0139] Using the position (x1', y1', z1') of the detection camera and the direction and / or angle at which the detection camera views the object from the position (x1', y1', z1'), a virtual first straight line passing through the position (x1', y1', z1') of the detection camera and the object can be calculated.

[0140] In a manner substantially the same as calculating the position (x1', y1', z1') of the detection camera and the virtual first straight line passing through the object, the position (x2', y2', z2') of the detection camera and the virtual second straight line passing through the object can be calculated using the coordinates (x2, y2) of the object and the position (x2', y2', z2') of the detection camera in the second image (313). The position (x3', y3', z3') of the detection camera and the virtual third straight line passing through the object, and the position (x4', y4', z4') of the detection camera and the virtual fourth straight line passing through the object can also be calculated substantially the same as in the above example.

[0141] For example, the electronic device (100) can calculate the intersection point of the first straight line to the fourth straight line. The electronic device (100) can determine the coordinates of the intersection point of the first straight line to the fourth straight line as the coordinates of the object.

[0142] The description of the operation of the above electronic device (100) to determine the coordinates of an object by using the coordinates (321, 323, 325, 327) of the object in the plurality of first images (311, 313, 315, 317) and the positions of the plurality of detection cameras (300a) is exemplary and is not limited to the above example. The electronic device (100) can determine the coordinates of an object by applying a known method for calculating the positions of the plurality of detection cameras (300a) in a set three-dimensional coordinate system or a method that can be easily derived by a person skilled in the art.

[0143] FIG. 5C is a diagram showing a plurality of first images (311, 313, 315, 317) received from a plurality of detection cameras (300a). Unlike FIG. 5B, FIG. 5C shows that the first images (311) and (313) include an object (411), but the first images (315) and (317) do not include an object (411). The electronic device (100) can determine the position of the object by using the coordinates (321, 323) of the object in the first images (311) and (313), and the positions of the detection cameras that captured the first images (311) and (313).

[0144] FIG. 5D is a diagram illustrating an operation of an electronic device (100) to determine the position of an object in time series. The electronic device (100) can determine the position of the object for each set frame. For example, when each of a plurality of detection cameras (300a) as in FIG. 5D captures an image at 8.3 ms / 120 fps, the electronic device (100) can determine the position of the object for each frame. The electronic device (100) can determine the position (541) of an object (e.g., a ping-pong ball) as (x1, y1, z1) in one frame. The electronic device (100) can determine the position (542) of an object (e.g., a ping-pong ball) as (x2, y2, z2) in two frames. The electronic device (100) can determine the position (543) of an object (e.g., a ping-pong ball) as (x3, y3, z3) in three frames. The electronic device (100) can determine the position (544) of an object (e.g., a ping-pong ball) as (x4, y4, z4) in the 4th frame. The electronic device (100) can determine the position (545) of an object (e.g., a ping-pong ball) as (x5, y5, z5) in the 5th frame. The electronic device (100) can determine the position (546) of an object (e.g., a ping-pong ball) as (x6, y6, z6) in the 6th frame. The electronic device (100) can determine the position (547) of an object (e.g., a ping-pong ball) as (x7, y7, z7) in the 7th frame. The electronic device (100) can determine the position (548) of an object (e.g., a ping-pong ball) as (x8, y8, z8) in the 8th frame.

[0145] The electronic device (100) can input the location of an object into a learned model (e.g., a second model) for each set frame (e.g., 1 frame). The electronic device (100) can generate a relay image using at least one second image among a plurality of second images received from a plurality of cameras for shooting, based on the output of the learned second model.

[0146] The method by which the electronic device (100) determines the position of the object is not limited to the examples illustrated in FIGS. 5A, 5B, 5C, and 5D. For example, the electronic device (100) can determine the three-dimensional coordinates of the object by using the distance between the plurality of detection cameras (300a) and the object and the positions of the detection cameras (300a). For example, the electronic device (100) can calculate the direction from the detection camera to the object by using the coordinates (321) of the object in the first image (321). The electronic device (100) can determine the position (e.g., three-dimensional coordinates) of the object within the three-dimensional coordinate system by using the positions of the detection cameras, the directions from the detection cameras to the object, and the distance between the detection cameras and the object.

[0147] The distance between the multiple detection cameras (300a) and the object can be calculated using various known methods. For example, the multiple capture cameras (300a) may be depth cameras. For example, if the multiple capture cameras (300a) are a single camera, various methods and / or algorithms for estimating the distance between the single camera and the object can be applied.

[0148]

[0149] FIG. 6 is a diagram illustrating an operation of an electronic device (100) generating data (630) according to various embodiments.

[0150] Referring to FIG. 6, an electronic device (100) according to one embodiment can determine data (630) by using at least one of the type of object, the location of the object, additional information, or a combination thereof.

[0151] According to one embodiment, the electronic device (100) can determine data (630) by using the position of object 1 (610-1) for each frame and the position of object n (610-n) for each frame. For example, the electronic device (100) can determine the speed of object 1 (610-1) by using the position of object 1 (610-1) within a set frame from the current point in time.

[0152] Referring to FIG. 6, the electronic device (100) can determine data (630) using the location of object 1 (610-1) for each frame, the location of object n (610-n) for each frame, and additional information (620). For example, the data (630) can include, but is not limited to, speed, pattern, path, and frequency.

[0153] For example, the electronic device (100) may determine additional information (620) using a plurality of first images. For example, the additional information (620) may include jersey numbers, (referee, player) gestures, information displayed on an electronic scoreboard, etc. For example, the electronic device (100) may output the additional information using a model (140). For example, the model (140) may be trained to identify objects and additional information (620) included in a plurality of input images.

[0154] According to one embodiment, the electronic device (100) may determine prediction data using the location of object 1 (610-1) for each frame, the location of object n (610-n) for each frame, additional information (620), and data (630). For example, the prediction data may include predicted information about an object or a game.

[0155] According to one embodiment, the electronic device (100) can input the location of object 1 (610-1) for each frame, the location of object n (610-n) for each frame, additional information (620) and data (630) into a learned prediction model to output prediction data.

[0156] According to one embodiment, the electronic device (100) may input the position of object 1 (610-1) for each frame, the position of object n (610-n) for each frame, additional information (620), data (630), and external information into a learned prediction model to output prediction data. For example, the external information may include information received from outside regarding objects and games (e.g., player information (height, age, ranking, win rate), opponent records, etc.).

[0157]

[0158] FIG. 7a, FIG. 7b, FIG. 7c, and FIG. 7d are diagrams showing data provided by an electronic device (100) according to various embodiments.

[0159] FIG. 7A is a drawing showing an image (710) displaying data provided by an electronic device (100). A user terminal (200) can display an image (710) displaying data using a display device of the user terminal (200).

[0160] As shown in Figure 7a, the data may include data about Player 1 in Match 1. The data may include the percentage of the area attacked by Player 1, and the average / maximum speed of the ping pong ball hit by Player 1.

[0161] The data shown in Fig. 7a is exemplary and is not limited to the example shown.

[0162] As shown in FIG. 7b, the electronic device (100) can determine a relay image (720) containing data.

[0163] For example, the electronic device (100) can generate an image (720) in which visualized data is combined with a relay image, as shown in FIG. 7b. The electronic device (100) can generate an image (720) by combining visualized data with a relay image.

[0164] For example, the data may include, but is not limited to, scoring rate / number of points, attack point percentage, serve point scoring rate / number of points, average speed, and top speed of Player 1's attack patterns (e.g., drives, backhand drives, cuts).

[0165] For example, the electronic device (100) can generate an image in which visualized data is overlapped with a relay image, such as the image (730) of FIG. 7c and the image (740) of FIG. 7d.

[0166] The electronic device (100) can generate an image (730) in which data visualized in a relay image (e.g., player 1 win probability, player 2 win probability, sub speed, and maximum speed in FIG. 7C) are overlapped, such as the image (730) in FIG. 7C.

[0167] The electronic device (100) can generate an image (740) in which data visualized in a relay image (e.g., the movement trajectory of a ping-pong ball in FIG. 7d) is overlapped, such as the image (740) in FIG. 7d.

[0168] For example, after a score is scored and before the players proceed to the next game, the electronic device (100) can determine the video (740). The electronic device (100) can determine the broadcast video (740) containing data based on the type of object (e.g., person, ping-pong ball) and location.

[0169]

[0170] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0171] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0172] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0173] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0174] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0175] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, at least one processor; and A memory electrically connected to at least one processor and storing at least one instruction executed by the processor. Including, At least one processor, When at least one of the above commands is executed, the electronic device receives a plurality of images captured with respect to a shooting space from a plurality of cameras; Identifying at least one object included in the plurality of images for each set frame; For each of the above-set frames, determine the position of at least one object in a three-dimensional coordinate system set with respect to the shooting space; Determining data related to the location of at least one object, To transmit the above data to the user terminal, Electronic devices.

2. In paragraph 1, At least one processor, Using each of the plurality of images, two-dimensional coordinates of at least one object are determined; Determining the position of at least one object using the two-dimensional coordinates of the at least one object, Electronic devices.

3. In paragraph 1, At least one processor, Recognize additional information using the above multiple images; determining said data based on the location of said at least one object and said additional information; Electronic devices.

4. In paragraph 1, At least one processor, Determining prediction data regarding the at least one object by using the location of the at least one object and at least one of the data, Electronic devices.

5. In paragraph 1, At least one processor, Identifying a relay image of the above shooting space and at least one object; Transmitting the above relay video and the above data to the user terminal, Electronic devices.

6. In paragraph 5, At least one processor, Receiving a plurality of second images for generating the relay image from the plurality of cameras; Generating the relay image as at least one image among the plurality of second images based on the type and location of the at least one object; Electronic devices.

7. In paragraph 5, At least one processor, Controlling at least one camera among the plurality of cameras based on the type and location of the at least one object; Electronic devices.

8. In paragraph 5, At least one processor, The above data generates a relay image visually overlapping the above relay image; Transmitting the above overlapped relay image to the user terminal, Electronic devices.

9. In paragraph 1, At least one processor, In response to a request received from the user terminal, transmitting the data to the user terminal, Electronic devices.

10. In terms of data analysis methods, An action of receiving multiple images captured from multiple cameras with respect to a shooting space; An operation of identifying at least one object included in the plurality of images for each set frame; For each of the above-set frames, an operation of determining the position of at least one object in a three-dimensional coordinate system set with respect to the shooting space; An operation of determining data related to the location of at least one object; and An action to transmit the above data to a user terminal including, Data analysis methods.

11. In paragraph 10, The operation of determining the position of at least one object is: An operation of determining two-dimensional coordinates of at least one object using each of the plurality of images; and An operation of determining the position of at least one object using the two-dimensional coordinates of the at least one object. including, Data analysis methods.

12. In paragraph 10, An operation of recognizing additional information using the above multiple images; Including more, The action of determining the above data is: determining said data based on the location of said at least one object and said additional information; Data analysis methods.

13. In paragraph 10, An operation of determining prediction data regarding at least one object by using at least one of the location of the at least one object and at least one of the data. including more, Data analysis methods.

14. In paragraph 10, An operation of identifying a relay image regarding the above shooting space and at least one object Including more, The action of transmitting to the above user terminal is: An operation of transmitting the above relay video and the above data to the user terminal. including, Data analysis methods.

15. In paragraph 14, The action of receiving the above relay video is: An operation of receiving a plurality of second images for generating the relay image from the plurality of cameras; and An operation of generating the relay image as at least one image among the plurality of second images based on the type and location of the at least one object. including, Data analysis methods.

16. In paragraph 14, An operation of controlling at least one camera among the plurality of cameras based on the type and location of at least one object. including more, Data analysis methods.

17. In paragraph 14, The operation of transmitting the above relay video and the above data to the user terminal is, An operation of generating a relay image in which the data is visually overlapped with the relay image; and An operation of transmitting the above overlapped relay image to the user terminal. including, Data analysis methods.

18. In paragraph 10, The action of transmitting to the above user terminal is: An operation of transmitting the data to the user terminal in response to a request received from the user terminal. including, Data analysis methods.

19. A computer program stored on a computer-readable recording medium for executing the method of any one of claims 9 to 18 in combination with hardware.

Images