Method for processing images acquired from an imaging device in conjunction with a computing device and system using the same

Abstract

According to some embodiments of the present disclosure, a method for processing an image, executed by a computing device including a processor, may include: acquiring an image; acquiring analysis information corresponding to an object included in the image from the image using an object analysis model; and acquiring characters included in the object from the analysis information corresponding to the object using an OCR model. A representative diagram may be shown in FIG. 2.

Description

[Technical field]

[0001] This disclosure discloses a method for processing an image acquired from a photographing device and a system using the same. Specifically, according to the method, a computing device acquires an overall image from a photographing device integrated with or linked to the computing device, detects one or more objects captured in the overall image, performs classification to calculate a category for each of the detected objects, and generates detailed classification information including characteristics and states of each of the objects as an analysis result for each of the objects. [Background technology]

[0002] A portable computing apparatus generally refers to a device equipped with a processor, a display, a microphone, and a speaker, some of which can be used as a portable terminal, which is a type of communication device. Conventionally, portable terminals have been designed so that user commands are input through input devices that require the user's touch, such as a keypad or a touch display, but with the development of voice recognition technology and computer vision technology, it has become possible to receive user commands from a distance using input devices such as a microphone and a camera, and to interact according to those commands.

[0003] However, portable computing devices do not have the ability to move around, and so their range of input and output is limited by their physical location.

[0004] As an example of the physical limitations of the input device mounted on the portable computing device, a camera, which is a non-contact input device, is mounted on the front or back of a mobile phone and has a limited field of view, so that in order to capture a desired image (picture), the user must hold the camera in his / her hand and directly change the composition (field of view (FOV)). As another example of the physical limitations mentioned above, the receiving sensitivity of a microphone mounted on a portable computing device decreases depending on the direction of the sound source.

[0005] Not only input but also output is limited by physical position. As an example, a touch display, which is an output device, generally has a shape close to a plane and is attached to some of the six faces of a mobile terminal, and a user holds the mobile terminal in his / her hand and uses the mobile terminal with the touch display of the mobile terminal facing his / her face. As another example, an infrared projector for acquiring a three-dimensional shape has a direction and angle of infrared light irradiation limited to a specific direction range of the mobile terminal, so that the user needs to hold the mobile terminal in his / her hand and adjust the direction of irradiation along a guide, as in the above example. In addition, Korean Patent Publication Nos. 10-2011-0032244, 10-2019-0085464, 10-2019-0074011, 10-2019-0098091, 10-2019-0106943, and 10-2018-0109499 have been proposed in this regard. Regarding this, (Non-Patent Document 1) Y. Zhou et al., "Learning to Reconstruct 3D Manhattan Wireframes From a Single Image," 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 7697-7706, doi: 10.1109 / ICCV.2019.00779., (Non-Patent Document 2) Shin, D., & Kim, I. (2018). Deep Image Understanding Using Multilayered Contexts. Mathematical Problems in Engineering, 2018, 1-11. https: / / doi.org / 10.1155 / 2018 / 5847460, (Non-Patent Document 3) Mo, K., Zhu, S., Chang, AX, Yi, L., Tripathi, S., Guibas, LJ, & Su, H. (2019).PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding. In 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. https: / / doi.org / 10.1109 / cvpr.2019.00100, (Non-Patent Document 4) Babaee, M., Li, L., & Rigoll, G. (2019). Person identification from partial gait cycle using fully convolutional neural networks. Neurocomputing, 338, 116 - 125., (Non-Patent Document 5) Muhammad, U. R., Svanera, M., Leonardi, R., & Benini, S. (2018). Hair detection, segmentation, and hairstyle classification in the wild. Image and Vision Computing, 71, 25 - 37. https: / / doi.org / 10.1016 / j.imavis.2018.02.001, (Non-Patent Document 6) Mougeot, G., Li, D., & Jia, S. (2019). A Deep Learning Approach for Dog Face Verification and Recognition. In PRICAI 2019: Trends in Artificial Intelligence (pp. 418 - 430). Springer International Publishing. https: / / doi.org / 10.1007 / 978-3-030-29894-4_34, (Non-Patent Document 7) Raduly, Z., Sulyok, C., Vadaszi, Z., & Zolde, A. (2018). Dog Breed Identification Using Deep Learning.In 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics(SISY).IEEE.https: / / doi.org / 10.1109 / sisy.2018.8524715,(Non-Patent Document 8)Wu, Z., Yao, T., Fu, Y., & Jiang, Y.-G.(2017).Deep learning for video classification and captioning.In Frontiers of Multimedia Research (pp. 3-29). ACM. https: / / doi.org / 10.1145 / 3122865.3122867, (Non-Patent Document 9) Wu, C.-Y., Girshick, R., He, K., Feichtenhofer, C., & Krahenbuhl, P. (2020). A Multigrid Method for Efficiently Training Video Models. In 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. https: / / doi.org / 10.1109 / cvpr42600.2020.00023, (Non-Patent Document 10) Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, SW (2018). Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features. IEEE Access, 6, 1155-1166. https: / / doi.org / 10.1109 / access.2017.2778011 has been proposed. . Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure aims to provide a proposal for overcoming the limitations of portable computing devices based on the aforementioned conventional technology, particularly with regard to the input of video (images), and proposes a technical method in which a gimbal is coupled to a video (image) capture device such as a camera that is linked to a portable computing device, and the computing device controls the gimbal to rotate the video (image) capture device around one or more axes, thereby enabling the video (image) input to be received and processed more dynamically.

[0007] The purpose of this disclosure is to solve the problems of the prior art and to present an image (image) processing method in a portable computing device that can recognize and track objects from video (images) captured by a camera or the like and actively obtain information related to the objects and the environment, and in particular, to grasp the relative position information and spatial information of an object in a coordinate system centered on the system of this disclosure based on the video (image), and to confirm the position of an object in space, thereby enabling long-distance input through the video (image). [Means for solving the problem]

[0008] To solve the above-mentioned problem, a method for processing an image is provided, executed by a computing device including a processor, the method may include: acquiring an image; acquiring, from the image, analysis information corresponding to an object contained in the image using an object analysis model; and acquiring, from the image, analysis information corresponding to the object using an OCR model, characters contained in the object.

[0009] Alternatively, the OCR model may perform the steps of: determining whether a character is displayed on a surface of the object; and, if it is determined that the character is displayed on the surface of the object, performing OCR on the character.

[0010] Alternatively, in paragraph 2, the step of performing the OCR on the character may include the steps of: extracting at least one video (image) sample from an object in which the character is displayed; determining a boundary line of an area in which the character is displayed from the at least one video (image) sample; generating a character display image (image) based on at least one of the boundary lines or boundary points that are points belonging to the boundary line from the at least one video (image) sample; and performing OCR on the character display image (image). Alternatively, the video (image) sample may be a video (image) pattern present at at least one of the boundary line or the boundary point, and the video (image) pattern may include at least one of a part of a character, a boundary of the character, a part of the character, or a background.

[0011] Alternatively, the step of generating the character display image (image) based on at least one of the boundary line or the boundary points which are points belonging to the boundary line from the at least one image (image) sample may include the steps of: obtaining an image (image) pattern located on at least one of the boundary line or the boundary points from the at least one image (image) sample as a boundary marker; and generating the character display image (image), which is an image (image) including the boundary marker, as a partial image (image) included in the image (image) of the object in which the character is displayed.

[0012] Alternatively, the boundary marker may include at least one of: a beginning marker, which is a video (image) pattern corresponding to the first character of the characters; or an end marker, which is a video (image) pattern corresponding to the last character of the characters.

[0013] Alternatively, the step of acquiring, as the boundary markers, a plurality of image (image) patterns located on at least one of the boundary lines or the boundary points from the at least one image (image) sample may include a step of acquiring, using a tracking controller, an image (image) of a character including the start marker and having a character recognition rate equal to or higher than a threshold value; and a step of determining whether the image (image) of the character includes the end marker.

[0014] Alternatively, the method may further include the step of: determining the image (image) of the text as the text display image (image) if the end marker is included in the image (image) of the text.

[0015] Alternatively, the method may further include the steps of: if the character image does not include the end marker, using the tracking controller to obtain an additional character image including a marker next to the last marker among the boundary markers included in the character image and having a character recognition rate equal to or greater than the threshold; combining the additional character image with the character image to generate a combined character image; and if the combined character image includes the end marker, determining the combined character image as the character display image.

[0016] Alternatively, the step of performing the OCR on the character may include the steps of: using a tracking controller to obtain an image of the object that includes the beginning of the character and has a character recognition rate above a threshold; performing the OCR on an area of ​​the first sentence in the image of the object; and performing natural language understanding (NLU) on a first text that is a primary result of the OCR, and calculating a first semantic value number that is a numerical value of semantic value.

[0017] Alternatively, the method may further include the step of: determining the first text as a result text that is a result of the OCR if the first semantic value value is equal to or greater than the threshold value.

[0018] Alternatively, the method may further include the steps of: performing the OCR on an area of ​​a sentence following the area of ​​the first sentence if the first semantic value value is less than the threshold; performing natural language understanding on a second text which is a primary result of the OCR, and calculating a second semantic value value which is a semantic value value; and, if the second semantic value value is greater than or equal to the threshold, calculating the second text as a result text which is the result of the OCR.

[0019] Alternatively, the computing device may be in conjunction with an imaging device and a gimbal, and the step of capturing the image may include using a tracking controller to actuate one or more rotation axes of the gimbal to control the direction of the imaging device; and using the tracking controller to capture the image by zooming in or zooming out of the imaging device.

[0020] A non-transitory computer-readable medium including a computer program for solving the above-mentioned problem, the computer program causing a computing device to execute a method for processing an image, the method may include: acquiring an image; acquiring, from the image, analysis information corresponding to an object contained in the image using an object analysis model; and acquiring, from the image, analysis information corresponding to the object using an OCR model, characters contained in the object.

[0021] In order to solve the above-mentioned problem, a computing device includes a processor and a communication unit, wherein the processor is capable of acquiring a video (image), and using an object analysis model to acquire from the video (image) analysis information corresponding to an object contained in the video (image), and using an OCR model to acquire characters contained in the object from the analysis information corresponding to the object. Effect of the Invention

[0022] According to exemplary embodiments of the present disclosure, it is possible to recognize and track one or more objects using video (images), to actively obtain information about the objects and the environment, in particular to obtain information about the state of an object from a long distance using video (images), to determine objects with which an object interacts using video (images), to obtain higher resolution detailed video (images) of portions of an object from a long distance, and to grasp from a long distance characters printed on an object or output using other means such as a display, thereby enabling long distance input using video (images) in a portable computing device. [Brief description of the drawings]

[0023] The following drawings attached for use in explaining the embodiments of the present invention are only a part of the multiple embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter referred to as "ordinary engineers") is able to derive other drawings based on these drawings without making any effort to conceive of the invention. [Figure 1] A conceptual diagram illustrating an example configuration of a computing device for executing a method for processing video (images) by a computing device (hereinafter referred to as a "video (image) processing method") based on one embodiment of the present disclosure. [Diagram 2]FIG. 1 is a conceptual diagram showing an example of the overall hardware and software architecture of a system for executing a video (image) processing method according to one embodiment of the present disclosure, including a computing device, a photographing device, and a gimbal. [Diagram 3] 1 is a flowchart illustrating an example of a video (image) processing method according to one embodiment of the present disclosure. [Figure 4] FIG. 2 is a block diagram showing an example of modules for performing each step of a video (image) processing method according to one embodiment of the present disclosure. [Diagram 5] FIG. 1 is a block diagram showing an example of a machine learning model used in multiple modules for a video (image) processing method according to one embodiment of the present disclosure. [Figure 6a] 1 is a flow chart showing, by way of example, several techniques that may be used to detect the floor (ground) plane of an object in the image processing method of the present disclosure. [Figure 6b] 1 is a flow chart showing, by way of example, several techniques that may be used to detect the floor (ground) plane of an object in the image processing method of the present disclosure. [Figure 6c] 1 is a flow chart showing, by way of example, several techniques that may be used to detect the floor (ground) plane of an object in the image processing method of the present disclosure. [Figure 6d] 1 is a flow chart showing, by way of example, several techniques that may be used to detect the floor (ground) plane of an object in the image processing method of the present disclosure. [Figure 7a] 1 is a diagram showing an example of object segmentation obtained by a video (image) processing method according to an embodiment of the present disclosure; [Figure 7b] 1 is an exemplary diagram for explaining multiple steps of detecting planes of two or more floor surfaces (ground surfaces) in an image (image) processing method based on one embodiment of the present disclosure. [Figure 7c]1 is a conceptual diagram for explaining a method for generating and using a reference plane circle and a measurement plane circle in a video (image) processing method based on one embodiment of the present disclosure. FIG. [Figure 8a] 1 is a flow chart showing, by way of example, several techniques that may be used to determine a target position in the image processing method of the present disclosure. [Figure 8b] 1 is a flow chart showing, by way of example, several techniques that may be used to determine a target position in the image processing method of the present disclosure. [Figure 8c] 1 is a flow chart showing, by way of example, several techniques that may be used to determine a target position in the image processing method of the present disclosure. [Figure 9a] 1 is a flowchart showing, by way of example, several methods used to control the orientation of a photographing device in the video (image) processing method of the present disclosure. [Figure 9b] 1 is a flowchart showing, by way of example, several methods used to control the orientation of a photographing device in the video (image) processing method of the present disclosure. [Figure 10a] 1 is a flowchart showing an example of a number of methods that may be used to perform OCR in the image processing method of the present disclosure. [Figure 10b] 1 is a flowchart showing an example of a number of methods that may be used to perform OCR in the image processing method of the present disclosure. [Figure 10c] 1 is a flowchart showing an example of a number of methods that may be used to perform OCR in the image processing method of the present disclosure. [Figure 11a] 1 is a diagram shown as an example to explain several ways of performing OCR in the image processing method of the present disclosure. [Figure 11b] 1 is a diagram shown as an example to explain several ways of performing OCR in the image processing method of the present disclosure. [Figure 11c]1 is a diagram shown as an example to explain several ways of performing OCR in the image processing method of the present disclosure. [Figure 11d] 1 is a diagram shown as an example to explain several ways of performing OCR in the image processing method of the present disclosure. BEST MODE FOR CARRYING OUT THEINVENTION

[0024] All prior documents cited in this disclosure are incorporated by reference in their entirety as if fully set forth in this disclosure. Unless otherwise defined, all terms used in this disclosure, including technical and scientific terms, shall be used as commonly understood by those of ordinary skill in the art. In addition, terms defined in common dictionaries shall be interpreted as having the same meaning as in the context of the relevant art, and shall not be interpreted as idealized or overly formal unless otherwise defined in this specification.

[0025] In the following detailed description of the present invention, in order to clearly show the objectives, technical solutions and advantages of the present invention, reference will be made to the accompanying drawings, which illustrate, by way of example, specific embodiments that can be implemented by the present invention. These embodiments will be described in detail to enable those of ordinary skill in the art to implement the present invention. When describing the present invention using the accompanying drawings, the same reference numerals will be used for the same components regardless of the reference numerals of the drawings, and duplicated descriptions thereof will be omitted.

[0026] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and they can be modified and embodied in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents, or alternatives that fall within the technical spirit.

[0027] Although terms such as first and second are used to describe various components, these terms should be construed only as distinguishing one particular component from another and in no way imply a particular order. Thus, a first component can be termed a second component, and similarly, a second component can be termed a first component.

[0028] When a component is described as being "connected" or "coupled" to another component, it should be understood that the component may be directly connected or connected to the other component, but there may also be other components in between.

[0029] The singular includes the plural unless the context clearly indicates otherwise. In this specification, the terms "comprise", "include", "have" or "having" are intended to indicate the presence of the stated features, numbers, steps, operations, components, parts, or combinations thereof, and are not to be understood as implying the exclusion of the possibility that one or more other features, numbers, steps, operations, components, parts, or combinations thereof may be present or added.

[0030] In addition, a "portion" or "part" of an object may mean only a part of the object, not the whole of the object, but should be understood to include the whole of the object unless otherwise specified in the context. A subset of a set is equivalent to a concept that includes the set itself.

[0031] In this disclosure, a "module" may mean hardware capable of performing the functions and operations of each name described in this disclosure, may mean computer program code capable of performing specific functions and operations, or may mean a recording medium in which computer program code capable of performing specific functions and operations is stored. In other words, a module may mean a functional and / or structural combination of hardware for executing the technical ideas in this disclosure and / or software for driving the hardware.

[0032] Strictly speaking, a "model" refers to a function configured to calculate output data from input data, as trained by machine learning. Such a "model" is a type of data structure or function that can be used by the aforementioned "modules."

[0033] However, since some ordinary engineers in the field of application of artificial intelligence have a habit of confusing the terms "module" and "model," in this disclosure document, the terms "module" and "model" may be used interchangeably, because such usage of the terms can be easily understood by ordinary engineers without confusing the respective concepts.

[0034] In this disclosure, the terms "training" and "learning" are terms that mean performing machine learning through computing along a process, and are not used to mean mental actions equivalent to human educational activities, as understood by ordinary engineers. In the field of statistics, the term "machine learning" is often used to refer to a series of processes to create a target function (f) that appropriately maps an input variable (X) to an output variable (Y). Calculating an output variable from an input variable using a target function is called "prediction," and "appropriate mapping" means that the difference between the true value and the predicted value is reasonably reduced. However, the reason for reasonably reducing the difference rather than minimizing it is that optimization may cause the so-called overfitting problem, that is, when actual data that deviates from the training data is applied, the prediction may not be successful, and appropriate empirical measures may be taken to resolve this problem.

[0035] In addition, in this disclosure, "inference" is a term that refers to the process of calculating output data from input data using a machine-learned model, but is particularly used to refer to the mechanical imitation of human mental processes. Similarly, in this disclosure, "analysis" by a machine is used to refer to the mechanical imitation of human mental processes, similar to inference.

[0036] In this disclosure, the term "Manhattan space" is used to refer to the same thing as that disclosed in the non-patent document Y. Zhou et al., "Learning to Reconstruct 3D Manhattan Wireframes From a Single Image," 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 7697-7706, doi:10.1109 / ICCV.2019.00779.

[0037] Furthermore, the present invention covers all possible combinations of the embodiments shown in this disclosure. It should be understood that the various embodiments of the present invention are different but not necessarily mutually exclusive. For example, a specific shape, structure, and characteristic described herein may be embodied in another embodiment without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components in each embodiment disclosed herein may be changed without departing from the spirit and scope of the present invention. Therefore, the following detailed description should not be taken in a limiting sense. In the drawings, similar reference symbols refer to the same or similar features in various aspects.

[0038] Unless otherwise indicated in this specification or clearly contradictory in the context, items indicated in the singular include the plural unless otherwise defined in the context. In addition, when describing the present invention, if it is determined that a specific description of a related known configuration or function may make the gist of the present invention unclear, the detailed description will be omitted.

[0039] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those skilled in the art to easily carry out the present invention.

[0040] FIG. 1 is a schematic diagram illustrating an example configuration of a computing device for executing a video (image) processing method according to one embodiment of the present disclosure.

[0041] Referring to FIG. 1, a computing device (100) according to one embodiment of the present disclosure includes a communication unit (110) and a processor (120), and is capable of directly or indirectly communicating with an external computing device (not shown) through the communication unit (110).

[0042] Specifically, the computing device (100) may use a combination of typical computer hardware (e.g., a computer; devices that may include processors, memory, storage, input and output devices, and other components of existing computing devices; electronic communication devices such as routers, switches, and the like; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause the computing device to function in a particular manner) to achieve the desired system performance. The storage may include storage devices such as hard disks and Universal Serial Bus (USB) memory, as well as network-based storage devices such as cloud servers. Here, the memory may be, but is not limited to, DDR2, DDR3, DDR4, SDP, DDP, QDP, magnetic hard disks, flash memory, and the like.

[0043] The communication unit (110) of the computing device as described above can transmit and receive requests and responses to other linked computing devices, such as mobile terminals, and as an example, such requests and responses can be made through the same TCP (Transmission Control Protocol) session, but are not limited to this, and can also be transmitted and received as, for example, UDP (User Datagram Protocol) datagrams.

[0044] Specifically, the communication unit 110 may be embodied in the form of a communication module including a communication interface. For example, the communication interface may include wireless Internet interfaces such as WLAN (Wireless LAN), WiFi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMax (World interoperability for Microwave access), HSDPA (High Speed ​​Downlink Packet Access), 4G, 5G, etc., and short-range communication interfaces such as Bluetooth (Bluetooth), RFID (Radio Frequency IDentification), Infrared Data Association (IrDA), UWB (Ultra-Wide Band), ZigBee, NFC (Near Field Communication), etc. In addition to the above, the communication interface may refer to any interface (e.g., a wired interface) that can communicate with the outside.

[0045] For example, the communication unit (110) can transmit and receive data to and from other computing devices via a suitable communication interface as described above. Furthermore, the communication unit (110) in a broad sense can include or be linked to an input unit such as a keyboard, mouse, touch sensor, or touch screen for receiving commands or instructions, an external input device such as a microphone, video camera, or LIDAR, radar, switch, button, joystick, or an external output device such as a sound card, a graphics card, a printing device, or a display, for example, a touch screen display unit, for receiving commands or instructions ... input device such as a sound card, a graphics card, a printing device, or a display, for example, a touch screen display unit, for receiving commands or instructions, or an external output device such as a sound card, a printing device, or a display unit, for example, a touch screen display unit. It is known that the computing device (100) can include a display device or be linked to an external display device through the communication unit (110) in order to display and provide an appropriate user interface to a user of the computing device, for example, a mobile terminal, and enable user interaction. For example, such a display device can be a touch screen capable of receiving touch input. The touch screen can detect an object, such as a finger or a stylus pen, that touches or approaches the display in a capacitive, inductive, or optical manner, and can determine the position of the detected object on the display.

[0046] The input device may also include a microphone. The type of microphone may include a dynamic microphone, a condenser microphone, etc., and a microphone having characteristics such as omnidirectional, unidirectional, and superdirectional may be used. A beamforming microphone and a microphone array may also be used, but are not limited to these. A microphone array refers to two or more microphones used to detect the direction of a sound source.

[0047] The output device may include a speaker, and the types of the speaker may include, but are not limited to, an omnidirectional speaker, a directional speaker, an ultra-directional speaker using ultrasonic waves, and the like.

[0048] The processor 120 of the computing device may include hardware configurations such as a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an ASIC, a CISC, a RISC, an FPGA, a SOC chip, or a tensor processing unit (TPU), a cache memory, and a data bus. The processor 120 may further include software configurations of an operating system and an application that performs a specific purpose. According to an embodiment of the present disclosure, the processor 120 may perform calculations for learning neural networks of various models. The processor 120 may perform calculations for learning neural networks, such as processing input data for learning in deep learning (DL), extracting feature values ​​from input data, calculating errors, and updating weights in the neural network using backpropagation. At least one of the CPU, a general purpose graphics processing unit (GPGPU), and / or a TPU of the processor 110 may process learning of network functions. For example, a CPU and a GPGPU may both train a network function or classify data using the network function. In one embodiment of the present disclosure, processors of multiple computing devices may be used together to train a network function or classify data using the network function. In one embodiment of the present disclosure, a computer program executed in a computing device (100) according to the present disclosure may be a program executable by a CPU, a GPGPU, or a TPU.

[0049] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network may be composed of a collection of interconnected computational units, generally called nodes. Such nodes may also be referred to as neurons. A neural network is composed of at least one or more nodes. The nodes (or neurons) that compose a neural network may be interconnected by one or more links.

[0050] In a neural network, one or more nodes connected via links can be in a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node that is an output node for a node can be an input node for another node, and vice versa. As mentioned above, the relationship between an input node and an output node can be established around a link. One or more output nodes can be connected to one input node via links, and vice versa.

[0051] In the relationship between an input node and an output node connected via a link, the value of the data of the output node can be determined based on the data input to the input node. In this case, the link interconnecting the input node and the output node can have a weight. The weight can be variable, and can be changed according to a user or an algorithm in order to execute a function required by the neural network. For example, when one or more input nodes are interconnected to one output node by respective links, the output node can determine the value of the output node based on the value input to the input node connected to the output node and the weight set to the link corresponding to each input node.

[0052] As described above, in a neural network, one or more nodes are interconnected through one or more links to form a relationship between an input node and an output node in the neural network. In a neural network, the characteristics of the neural network can be determined by the number of nodes and links, the correlation between the nodes and links, and the weight value assigned to each link. For example, if there are two neural networks that have the same number of nodes and links but different link weight values, the two neural networks can be recognized as different.

[0053] A neural network can be composed of a set of one or more nodes. A subset of the nodes in the neural network can be organized into a layer. A portion of the nodes in the neural network can be organized into a layer based on the distance from a first input node. For example, a set of nodes whose distance from a first input node is n can be organized into an n-layer. The distance from the first input node can be defined based on the minimum number of links that must be traversed to reach the node from the first input node. However, this definition of a layer is given arbitrarily for the convenience of explanation, and the position of a layer in a neural network can be defined in a manner different from that described above. For example, the layer of a node can be defined based on the distance from a final output node.

[0054] The first input node may refer to one or more nodes in a neural network to which data is directly input without passing through a link in relation to other nodes. Alternatively, the first input node may refer to a node in a neural network that does not have other input nodes connected via links in relation to nodes based on links. Similarly, the final output node may refer to one or more nodes in a neural network that do not have an output node in relation to other nodes. Moreover, the hidden node may refer to a node that does not correspond to the first input node or the final output node and that constitutes a neural network.

[0055] The neural network in one embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is the same as the number of nodes in the output layer, and the number of nodes decreases once and then increases again as the input layer progresses from the hidden layer. The neural network in another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is smaller than the number of nodes in the output layer, and the number of nodes decreases as the input layer progresses from the hidden layer. Furthermore, the neural network in another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is greater than the number of nodes in the output layer, and the number of nodes increases as the input layer progresses from the hidden layer. The neural network in another embodiment of the present disclosure may be a neural network in which the above-mentioned neural networks are combined. A deep neural network (DNN) can refer to a neural network including multiple hidden layers in addition to an input layer and an output layer. A deep neural network can be used to grasp the latent structures of data. In other words, it is possible to grasp the latent structures of photos, text, video, audio, and music (for example, what objects are in the photo, what are the contents and emotions of the text, what are the contents and emotions of the audio, etc.). A deep neural network can include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a restricted boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, a generative adversarial network (GAN), and the like. The above-mentioned deep neural networks are merely examples, and the present disclosure is not limited thereto.

[0056] Neural networks can be trained using at least one of supervised, unsupervised, semisupervised, or reinforcement learning. Training a neural network can be a process of applying knowledge to the neural network to make it perform a particular operation. A neural network can be trained in a direction that minimizes the error of the output. In training a neural network, training data is repeatedly input to the neural network, the error of the neural network output and the target regarding the training data is calculated, and the error of the neural network is back-propagated from the output layer of the neural network to the input layer of the neural network as a direction to reduce the error, and the weight of each node of the neural network is updated. In the case of supervised learning, training data in which the correct answer is labeled is used for each piece of training data (i.e., labeled training data), and in the case of unsupervised learning, the correct answer may not be labeled for each piece of training data. That is, for example, the training data in supervised learning for data classification can be data in which each piece of training data is labeled with a category. The labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the training data, which is the input, with the output of the neural network. The calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer) in the neural network, and the connection weights of each node in each layer of the neural network can be updated through backpropagation. The amount of change in the connection weights of each node to be updated can be determined by a learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The method of applying the learning rate can change depending on the number of iterations of the neural network learning cycle. For example, in the early stage of neural network learning, the learning rate can be increased to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, and in the later stage of learning, the learning rate can be decreased to increase accuracy.

[0057] In the training of a neural network, the training data can generally be a subset of the actual data (i.e., data to be processed using the trained neural network), and thus there may be a training cycle in which the error in the training data decreases but the error in the actual data increases. Overfitting is a phenomenon in which the training data is over-trained in this way, resulting in an increase in error in the actual data. For example, a neural network that has learned about cats by looking at a yellow cat cannot recognize that it is a cat when it sees a cat of a color other than yellow, which can be a type of overfitting. Overfitting can cause an increase in errors in machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout, which deactivates some of the nodes of the network during the training process, and the use of a batch normalization layer can be applied.

[0058] In one embodiment of the present disclosure, a computer-readable medium having a data structure stored thereon is disclosed.

[0059] A data structure can refer to the organization, management, and storage of data that allows efficient access and modification of the data. A data structure can refer to the organization of data to solve a specific problem (e.g., data retrieval, data storage, data modification in the shortest time). A data structure can also be defined as a physical or logical relationship between multiple data elements designed to support a specific data processing function. A logical relationship between multiple data elements can include a linking relationship between multiple user-defined data elements. A physical relationship between multiple data elements can include an actual relationship between multiple data elements that is physically stored in a computer-readable storage medium (e.g., a permanent storage device). A data structure can specifically include a collection of data, a relationship between the data, and a function or instruction that can be applied to the data. By utilizing an effectively designed data structure, a computing device can perform operations while minimizing the use of computing device resources. Specifically, a computing device can increase the efficiency of operations, reading, retrieval, deletion, comparison, exchange, and search through an effectively designed data structure.

[0060] Data structures can be classified into linear data structures and non-linear data structures based on the shape of the data structure. A linear data structure can be a structure in which one piece of data is connected to one piece of data. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a collection of a series of data that has an internal order. A list can include linked lists. A linked list can be a data structure in which data is linked in a manner in which each piece of data has a pointer and is connected in a line. In a linked list, the pointer can include information regarding the connection to the next piece of data or the previous piece of data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circularly linked list depending on the shape. A stack can be a data array structure in which access to data is restricted. A stack can be a linear data structure in which data can be manipulated (e.g., inserted or deleted) only at one end of the data structure. Data stored in a stack can be a LIFO (Last in First Out) data structure. A queue is also a data structure that limits access to data, but unlike a stack, it can be a first-in, first-out data structure (FIFO). A deck can be a data structure that allows data to be processed at both ends of the data structure.

[0061] A nonlinear data structure can be a structure in which multiple data are connected after one piece of data. A nonlinear data structure can include a graph data structure. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. A graph data structure can include a tree data structure. A tree data structure can be a data structure in which there is one path connecting two different vertices among multiple vertices included in the tree. In other words, the graph data structure can be a data structure that does not form a loop.

[0062] Throughout this specification, the terms "arithmetic model," "neural network," "network function," and "neural network" can be used with the same meaning. Hereinafter, they will be unified and described as "neural network." The data structure can include a neural network. The data structure including the neural network can be stored in a computer-readable medium. The data structure including the neural network can include data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, and the like. The data structure including the neural network can include any of the components disclosed above. That is, when a data structure including a neural network is configured, it can be configured to include all of data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc., or any combination thereof. In addition to the above configuration, the data structure including a neural network can include any other information that determines the characteristics of the neural network. In addition, the data structure can include any form of data used or generated in the computation process of the neural network, and is not limited to the above. The computer-readable medium can include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network can be configured of a set of interconnected computation units, generally called nodes. Such nodes can also be called neurons. A neural network is configured to include at least one or more nodes.

[0063] The data structure may include data to be input to the neural network. The data structure including the data to be input to the neural network may be stored on a computer-readable medium. The data to be input to the neural network may include training data input during the training process of the neural network and / or input data to the neural network after training has been completed. The data to be input to the neural network may include data that has been pre-processed and / or data that is the target of pre-processing. Pre-processing may include a data processing process for inputting data to the neural network. Thus, the data structure may include data that is the target of pre-processing and data that is generated by pre-processing. The above data structures are merely examples, and the present disclosure is not limited thereto.

[0064] The data structure may include weights of the neural network. (In this specification, weights and parameters may be considered to have the same meaning.) The data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include multiple weights. The weights may be variable, but may be changed by a user or an algorithm to perform a function required by the neural network. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine the value of data output from the output node based on multiple values ​​input to the input nodes connected to the output node and weights set to the links corresponding to each input node. The above data structures are merely examples, and the present disclosure is not limited thereto.

[0065] By way of example and not limitation, the weights may include weights that are variable during the training of the neural network and / or weights at which the neural network has been trained. The weights that are variable during the training of the neural network may include weights at the start of a training cycle and / or weights that change during a training cycle. The trained weights of the neural network may include weights at which the training cycle has been completed. Thus, a data structure including weights of a neural network may include a data structure including weights that are variable during the training of the neural network and / or weights at which the neural network has been trained. Thus, the weights and / or combinations of each of the weights described above are intended to be included in a data structure including weights of a neural network. The aforementioned data structures are merely exemplary and the disclosure is not limited thereto.

[0066] The data structure including the neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) through a serialization process. Serialization can be a process of converting the data structure into a form that can be stored in the same or a different computing device and later reconstructed for use. The computing device can serialize the data structure and transmit and receive data over a network. The serialized data structure including the neural network weights can be reconstructed in the same or another computing device through deserialization. The data structure including the neural network weights is not limited to serialization. Furthermore, the data structure including the neural network weights can include data structures for increasing the efficiency of calculations while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in nonlinear data structures). The above items are merely examples, and the present disclosure is not limited thereto.

[0067] The data structure may include hyper-parameters of the neural network. The data structure including the hyper-parameters of the neural network may be stored in a computer-readable medium. The hyper-parameters may be variables that vary depending on the user. The hyper-parameters may include, for example, a learning rate, a cost function, a number of learning cycle iterations, weight initialization (e.g., setting a range of weight values ​​to be subjected to weight initialization), and a number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structures are merely examples, and the present disclosure is not limited thereto.

[0068] FIG. 2 is a conceptual diagram showing an example of the overall hardware and software architecture of a system for performing an image processing method according to one embodiment of the present disclosure, including a computing device, a camera device, and a gimbal.

[0069] 2, the configuration of the method and device according to the present invention will be described. The computing device (100) may include a photographing device (200) and may be connected to an external photographing device (200) wirelessly or by wire, and the computing device (100) may be connected to a gimbal (300) wirelessly or by wire, or may include the gimbal (300) that performs the function of controlling the attitude of the photographing device (200). In order to control the attitude of the photographing device (200), the gimbal (300) may include the photographing device (200) or may be provided with a predetermined mechanism (e.g., an adsorption plate, etc.) to which the photographing device (200) can be fixed.

[0070] For the above-mentioned attitude control, the gimbal (300) can have one or more rotation axes, examples of which are described in Published Patent Application No. 10-2019-0036323. The gimbal (300) can actively improve the input range of the image capture device (200) by controlling the attitude.

[0071] If the gimbal (300) has one axis of rotation, that axis of rotation can be the yaw axis (Y), which allows the imaging device (200) to interact with objects in space with minimal rotation.

[0072] If the gimbal (300) has two rotation axes, the rotation axes can be the yaw axis and the pitch axis (P), and if the gimbal (300) has three rotation axes, the rotation axes can be the yaw axis, the pitch axis, and the roll axis (R).

[0073] The gimbal (300) may include a power supply unit (310) as a hardware component thereof. The power supply unit (310) may receive external power by wire or wirelessly and by DC or AC. The power supplied to the power supply unit (310) may be used in the gimbal (300) or the computing device (100). The power supply unit (310) may be used to charge a battery built in the gimbal (300) or a battery built in the computing device.

[0074] The gimbal (300) may also include at least one gimbal motor (330; not shown) as a hardware component. Each of the gimbal motors (330) is configured to change the orientation of the image capture device (200) or the computing device (100) in which the image capture device (200) is built along the aforementioned rotation axis, and the gimbal motor (330) may be, but is not limited to, a DC motor, a step motor, or a brushless motor. The gimbal (300) may further include a gear for converting the torque of the motor in addition to the gimbal motor (330).

[0075] The motors (330) of the gimbal (300) are intended to point the imaging device (200) or computing device (100) fixed to the gimbal in the direction of a particular object, and it is preferable that the rotational axes of the motors (330) are arranged parallel to the yaw, pitch, roll, and other axes of the imaging device (200) or computing device (100), but a person of ordinary skill in the art will easily understand that there is no reason to be limited to this.

[0076] The gimbal (300) may further include at least one sensor (340; not shown) as a hardware component thereof. The sensor (340) may detect one or more of the following with respect to a fixed portion of the gimbal (300) or the motor (330): position, angular position, displacement, angular displacement, velocity, angular velocity, acceleration, and angular acceleration. The types of the sensor (340) include, but are not limited to, an acceleration sensor, a gyro sensor, a magnetic sensor such as a geomagnetic sensor, a Hall sensor, a pressure sensor, an infrared sensor, a proximity sensor, a motion sensor, a light fading sensor, an image (image) sensor, a GPS sensor, a temperature sensor, a humidity sensor, an air pressure sensor, and a LIDAR sensor.

[0077] Sensors (340) that cannot be mounted on the computing device (100) due to weight and volume limitations, especially in portable computing devices, can be mounted on the gimbal (300) and can be used to obtain information about the surroundings of the gimbal (300).

[0078] Specific functions and effects of the present invention that can be achieved by the individual components outlined in FIG. 2 will be described in detail below with reference to FIG. 3 to FIG. 11d. For ease of explanation, the components shown in FIG. 2 are exemplified as being implemented in one computing device, but it is obvious that the computing device (100) that executes the method of the present disclosure may be configured to be configured to cooperate with one another. For example, the gimbal (300) may be configured as an independent computing device, and the gimbal (300) and the computing device (100), for example, a portable computing device such as a portable terminal, may cooperate with one another. In this case, the gimbal (300) may instead execute at least some of the functions executed by the portable terminal (100). In other words, a person skilled in the art may configure multiple devices to cooperate with one another and execute the method of the present disclosure in various ways.

[0079] Fig. 3 is a flow chart showing an example of a video (image) processing method according to an embodiment of the present disclosure, Fig. 4 is a block diagram showing an example of modules that execute each step of the video (image) processing method according to an embodiment of the present disclosure, and Fig. 5 is a block diagram showing an example of a machine learning model used in a plurality of modules for the video (image) processing method according to an embodiment of the present disclosure.

[0080] Referring to FIG. 3, the video (image) processing method according to the present disclosure first includes a video (image) acquisition step (S1000) in which a video (image) input module (4100) implemented by a computing device (100) acquires an entire video (image) from a photographing device (200) included in the computing device (100) or linked through a communication unit (110) of the computing device (100).

[0081] Here, "the entire video (image)" refers to a video (image) that is contrasted with a video (image) that is a part of the entire video (image), such as an object video (image) described below.

[0082] Next, the image processing method further includes a category classification step (S2000) in which an object analysis module (4200) implemented by a computing device (100) detects one or more objects appearing in the entire image and performs classification to calculate a category for each of the detected objects.

[0083] Here, the category of an object refers to the result of classifying an object into a person, a tree, a puppy, and so on.

[0084] In a category classification step (S2000), concurrently with the classification, the position of each of one or more objects can be calculated from the overall image as part of the two-dimensional measurement.

[0085] The difference between the 2D measurement mentioned above and the 3D measurement described below is that 2D measurement is performed based on the 2D coordinate system shown in the video (image) without considering information related to three-dimensional depth (or perspective), while 3D measurement is a measurement that considers not only the 2D coordinates in the video (image) but also information related to depth (or perspective).

[0086] Specifically, the category classification step (S2000) may include a step (S2100) of resizing the entire image (image) to a resolution lower than the original resolution of the entire image (image), and a step (S2200) of inputting the resized image (image) into an object analysis model (M420) and calculating the category, position, and importance of each of the objects.

[0087] For example, the processor 120 of the computing device 100 may use the object analysis model 420 to obtain, from the video, analysis information corresponding to objects contained in the video. The analysis information may include at least one of classification information indicating a category of the object, location information indicating a location of the object, and / or importance information indicating a priority of the object in the video.

[0088] The object analysis model (M420) is a model for analyzing objects contained in a video (image), and can include an object classification model, a localization model, an object detection model, a segmentation model, etc.

[0089] According to one embodiment of the present disclosure, the processor (120) can classify the type (class) of an object in an input video (image) using an object analysis model (M420) that performs object classification. For example, if a person is shown in the input video (image), the processor (120) can obtain an output of "the type of the input video (image) is a person" for the input video (image) using the object analysis model (M420). The above items are merely examples, and the present disclosure is not limited thereto.

[0090] According to an embodiment of the present disclosure, the processor 120 may also output location information indicating where an object in an image is located in the image using an object analysis model M420 that performs classification and regionalization. For example, when the processor 120 uses an object analysis model M420 that performs classification and regionalization, the processor 120 may recognize an object in the image using a bounding box and output location information. The bounding box may transmit location information of the object by outputting the coordinates of the left, right, top, and bottom of the box. The above items are merely examples, and the present disclosure is not limited thereto.

[0091] In accordance with one embodiment of the present disclosure, the processor (120) may detect at least one object using an object analysis model (M420) for object detection. The processor (120) may detect multiple objects and extract location information by simultaneously classifying and localizing at least one object using the object analysis model (M420) for object detection. The foregoing is by way of example only, and the present disclosure is not limited thereto.

[0092] In accordance with one embodiment of the present disclosure, the processor 120 may classify pixels using a segmentation object analysis model M420 to detect objects by distinguishing the boundaries of objects in an image from the background. The foregoing is by way of example only, and the present disclosure is not limited thereto.

[0093] It is well known to those of ordinary skill in the art that the resizing performed in step (S2100) is intended to reduce the amount of calculations performed by the object analysis model (M420) and to improve the processing speed of the object analysis module (4200).

[0094] The importance resulting from step (S2200) can be used as a measure to determine the priority of the objects, or conversely, high priority objects can be given high importance and / or low priority objects can be given low importance. Priority will be described in more detail below.

[0095] For objects whose importance is equal to or greater than a predetermined threshold, the computing device 100 can extract and sample an object image, which is an image of the object, from the entire image (crop feed step S2500). Since in step S2200 a resized image is used and some of the data of the entire image is lost, the crop feed step S2500 is intended to enable the use of image information before loss for objects with relatively high importance.

[0096] Continuing to refer to FIG. 3, the image processing method according to the present disclosure further includes a detailed classification step (S3000) in which a detailed classification module (4300) implemented by a computing device (100) generates detailed classification information including characteristics and status of each detected object as an analysis result for the object.

[0097] Here, an object is a concept that includes a spatial object, which is an object that corresponds to the "space" in which the entire video (image) was shot, and the detailed classification information of the spatial object can include information related to the space. Object characteristics and states

[0098] The properties of an object refer to the characteristics of the object that are generally time invariant, whereas the state of the object refers to the characteristics of the object that are generally time variable. Specifically, the characteristics of the object may include information of partial objects that are components that form part of the object or belong to the object. For example, if a person is detected as an object from an entire video (image), the person's arms, legs, eyes, and other parts, as well as the clothes and footwear worn by the person, are partial objects of the object.

[0099] In order to detect partial objects as described above, the detailed classification step (S3000) may include a step of attempting to detect partial objects that are part of the object or that are components belonging to the object (S3920); and if the partial object is detected, a step of additionally generating analysis results of the characteristics and conditions related to the partial object as part of the detailed classification information (S3940).

[0100] The characteristics of the object may also include at least one of a main color of the object, a sub-sub-object which is information pointing to the partial object of the object, a subject which is information pointing to the other object if the object is a partial object of the other object, a size of the object, one or more materials of the object including a main material of the object, a transparency of the object, text shown on the surface of the object, and whether the object can move on its own.

[0101] Here, the size of the object can be a size measured by two-dimensional or three-dimensional measurement, and the transparency of the object is a property that the object can have when it has a transparent part such as a glass window, and for example, an opaque object can have a value of 0, and an object made of a transparent material such as glass can have a positive value.

[0102] On the other hand, the state of the object may include at least one of the position of the object, the attitude of the object, the behavior of the object, the direction of the object, whether the object is in contact with the floor (ground), and the velocity of the object.

[0103] Here, the position of the object can be a position measured by 2D or 3D measurement, the pose of the object can be inferred from the position information of partial objects of the object, and the behavior of the object can be inferred from the poses in succession in time.

[0104] Also, the orientation of the object can be inferred from the position information of the object or the behavior of the object.

[0105] Whether or not the above-mentioned object is in contact with the floor (ground) refers to whether or not the above-mentioned object is in contact with the plane of the floor (ground) of the spatial object to which the object belongs. Examples of objects that have a true value include chairs, desks, utility poles, and car tires.

[0106] The detailed classification information of the object may further include attributes indicating information related to input and output of the object in the system, in addition to the characteristics and states. The attributes of the object may include at least one of a data input time including a time when raw data of the object was first input, and an operation authority related to the system based on the present disclosure that is given to the object.

[0107] In order to generate detailed classification information of the object as described above, the detailed classification step (S3000) may include a step (S3200) of selecting a detailed classification model (M430), which is a collection of models composed of at least one model pre-trained to match the individual category to which the individual object belongs, for an individual object that is an object whose importance is a predetermined second numerical value or more, so as to obtain the characteristics and the state of each of objects corresponding to the individual category to which the individual object belongs; and a step (S3400) of inputting an individual object image (image), which is a video (image) of the individual object, into the selected detailed classification model (M430) and generating an object record including the detailed classification information as an object record belonging to the individual object according to the identifier and the identifier.

[0108] Here, the detailed classification model (M430) is for classifying one or more objects. In other words, as detailed classification information generated by the detailed classification model (M430), it is possible to assign an identifier to each of a plurality of objects, which can classify each of the objects.

[0109] Also, here, an object record refers to a record that includes information belonging to the identifier of each object. Examples of information belonging to the identifier of each object may include, if the object is a person, the person's face shape, height, walking style, tattoos, hairstyle, etc., and if the object is a dog, the dog's head shape, fur pattern and color, breed, etc. The object record may include information of other objects that are owned by each object, which may be identifiers of the other objects.

[0110] Among these, in particular, it is possible to classify a person's walking style using artificial intelligence methodologies, as described, for example, in the non-patent literature paper Babaee, M., Li, L., & Rigoll, G. (2019). Person identification from partial gait cycle using fully convolutional neural networks. Neurocomputing, 338, 116-125.

[0111] Also, it is possible for a person's hairstyle to be classified using artificial intelligence methodologies, as described, for example, in the non-patent literature article Muhammad, U. R., Svanera, M., Leonardi, R., & Benini, S. (2018). Hair detection, segmentation, and hairstyle classification in the wild. Image and Vision Computing, 71, 25-37. https: / / doi.org / 10.1016 / j.imavis.2018.02.001.

[0112] However, one of ordinary skill in the art will appreciate that unexpected information disclosed in the prior art documents, such as those mentioned above, may also be obtained through artificial intelligence methodologies.

[0113] The step (S3200) of selecting a detailed classification model (M430) for each individual object can be performed by a classification model selection module (4320) embodied by the computing device (100). The classification model selection module (4320) performs the function of selecting, after the category of an object has been obtained, a detailed classification model adapted to the category, together with an algorithm to be applied to the detailed classification model.

[0114] For example, if the object category is a person, the classification model selection module (4320) can select a detailed classification model that generates detailed classification information that can identify a person, such as a person's facial shape, height, walking style, tattoos, hairstyle, etc., and if the object category is a dog, it can select a detailed classification model that generates detailed classification information that can identify a dog, such as a dog's head shape, fur pattern and color, breed, etc.

[0115] The fact that not only human face shapes but also dog head shapes can be classified using artificial intelligence methodologies is described, for example, in the non-patent literature Mougeot, G., Li, D., & Jia, S. (2019). A Deep Learning Approach for Dog Face Verification and Recognition. In PRICAI 2019: Trends in Artificial Intelligence (pp. 418-430). Springer International Publishing. https: / / doi.org / 10.1007 / 978-3-030-29894-4_34.

[0116] For information on artificial intelligence methodologies for classifying dog breeds, see the non-patent literature Raduly, Z., Sulyok, C., Vadaszi, Z., & Zolde, A. (2018). Dog Breed Identification Using Deep Learning. In 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics (SISY). IEEE. https: / / doi.org / 10.1109 / sisy.2018.8524715.

[0117] On the other hand, since the detailed classification model (M430) is a collection of models, it can include a measurement model (M431) that calculates at least one of the following by performing at least one of two-dimensional and three-dimensional measurements of the object: the position of the object, whether the object is in contact with the floor (ground), the direction of the object, the speed of the object, the attitude of the object, and / or the size of the object. Here, the size of the object can include at least one of the following one-dimensional dimensions: height, width, and / or depth (or depth), two-dimensional dimensions including the surface area of ​​the object, and / or three-dimensional dimensions including the volume of the object.

[0118] For example, the volume of an object may be calculated based on at least one (one or more) of an object segmentation, an object pose and / or depth information of the object.

[0119] When the object is a spatial object, the measurement model (M431) can calculate at least one of the following: a system position that is the origin of at least one of the orientations and coordinates of the spatial object, a Manhattan space that is a volumetric space that includes objects included in the space of the spatial object, a plane of the floor surface (ground) detected from the space, a gravity vector applied to the space, an empty volumetric space excluding the objects included in the space from the space, a partial object of the spatial object, and a direction of the space. The space can be indoors or outdoors.

[0120] Here, the partial object of a spatial object refers to a partial object that constitutes the space of the spatial object, and the partial object is fixed in the space. Examples of the partial object of the spatial object may include a glass window, a door, a wall, a kitchen, a road, a footbridge, etc.

[0121] System location refers to the location of the system according to this disclosure that serves as an origin of orientation or coordinates in an indoor or outdoor space.

[0122] Also, the spatial direction refers to the direction of an object that is the spatial object, but can be relative to a portion of the system or space of this disclosure.

[0123] The depth information in the 3D measurements can be predicted using artificial intelligence methodologies that derive depth information from the image or can be provided as supplement to the image from other sensors (340) such as lidar, radar, ultrasonic sensors, etc.

[0124] Specifically, in step (S3400), the three-dimensional measurement may further include a process (S3410) of identifying a length reference object, which is an object that satisfies the condition that the deviation in length of at least a portion of the object, which is a partial object that constitutes the object or a part of the object, or is a component belonging to the object, is smaller than a predetermined standard, measuring the two-dimensional length of the length reference object, and a process (S3420) of detecting the plane of the floor surface (ground) of the object.

[0125] The purpose of this 3D measurement is to use information from one or more objects to determine the relative position between the system of this disclosure and the objects and / or the absolute position of the objects.

[0126] For example, the height of an adult male, an example of a length reference object, can be used as a reference length for objects belonging to other categories, such as doors, pencils, cups, etc., but can also be used to measure distance.

[0127] As another example of a length reference object, when multiple doors are detected in an entire image or space (or spatial object), doors of the same design in a space have roughly the same height and can therefore be used as length reference objects for distance determination.

[0128] Partial objects can also be used as length reference objects if the deviation in the length of the partial object is relatively small; for example, the horizontal length of a human eyeball can be used as a length reference object because the standard deviation is relatively small.

[0129] In the process (S3410), when measuring the two-dimensional length of the length reference object, if the orientation information of the length reference object is available, it is possible to calculate a corrected length by reflecting the tilt due to the orientation of the length reference object. The orientation of the object and the calculation thereof will be described later.

[0130] Floor (ground) plane There are various possible methods for detecting the plane of the floor (ground) of the object in process (S3420), for example, a method for detecting the plane of the floor (ground) that is in contact with the object along the vector direction of gravity acting on the object may be used.

[0131] In addition, the detection of the floor (ground) plane of an object may be performed using a method that uses Manhattan space detection; if there are two or more length reference objects, a floor (ground) plane is generated between the length reference objects and then expanded; and if the length reference object is moved, a floor (ground) plane is generated between the length reference object before the movement and the length reference object after the movement and then expanded.

[0132] 6a-6d are flow charts showing, by way of example, several techniques that may be used to detect the floor (ground) plane of an object in the image processing method of the present disclosure.

[0133] 6a, a first specific embodiment of detecting the floor (ground) plane of an object (S3420) starts with a step of detecting at least one size-equivalent object (S3422a) that belongs to a size-equivalent category, which is a category that satisfies a condition that the deviation in size of at least a part of an object or a partial object of the object that belongs to a certain category is smaller than a predetermined standard. Here, the size can be one or more of width, height, and depth (or depth).

[0134] For example, a category that meets the condition that the deviation in size is less than a given standard may be the category of desks. A particular type of desk has a relatively small deviation in height. If the height of a certain type of desk is, on average, 70-74 cm, the standard length of an object of that size may be 72 cm.

[0135] Optionally, the size-equivalent object may be a floor-based object. A floor-based object is an object used to detect the plane of the floor, and at least a part of the object segmentation is generally in contact with the floor. For example, a desk and a chair are such floor-based objects.

[0136] Following step (S3422a), the first embodiment of the process (S3420) includes a step (S3424a) of detecting floor (ground) contact points, which are contact points of the size-equivalent objects to the bottom of the object segmentation.

[0137] FIG. 7a is a diagram showing an example of object segmentation obtained by the video (image) processing method of the present disclosure.

[0138] Methods for obtaining object segmentation are known to the ordinary skilled artisan, but it is possible to refer, for example, to the non-patent literature article Mo, K., Zhu, S., Chang, AX, Yi, L., Tripathi, S., Guibas, LJ, & Su, H. (2019). PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding. In 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE.

[0139] 7a, in the object segmentation of a chair given as an example, the bottom of the region (712, 714, 716, 718) corresponding to the legs of the chair can be considered to be in contact with the floor (ground), so it is possible to detect a floor (ground) contact point (720) that is a contact point with the bottom of the chair. As another example, the bottom of the object segmentation of a door comes in contact with the floor (ground), so it is possible to detect a floor (ground) contact point that is a contact point with the door.

[0140] Following step (S3424a), the first embodiment of the process (S3420) includes step (S3426a) of determining a three-dimensional distance of the size-comparable object based on the two-dimensional lengths of the size-comparable object. For example, a chair located farther away has at least one (or more) of two-dimensional length (height), width and / or depth (or depth) smaller than a chair located closer, and this can be used to determine the three-dimensional distance.

[0141] After step (S3426a), the first embodiment of the process (S3420) further includes step (S3428a) of connecting the floor (ground) contact points (720) to generate a three-dimensional reference plane (730) and extending the three-dimensional reference plane to generate a floor (ground) plane (740). According to Fig. 7a, a three-dimensional reference plane can be generated by connecting the bottom sections of the chair legs, which can be further extended to the floor (ground) plane (740).

[0142] Here, the extension from the three-dimensional reference plane is performed up to the boundaries of the corners of the gravity horizontal object, the start point of the wall, the bottom of the gravity horizontal object, etc., and the result is the plane of the floor surface (ground). Here, the gravity horizontal object generally refers to an object arranged to stand upright parallel to the direction of gravity, for example, a glass window and a wall surface. Specifically, the gravity horizontal object can be an object (e.g., a door, a window, etc.) having lines and / or faces arranged along the horizontal direction with respect to the gravity vector. The gravity horizontal object can be an object (e.g., a door frame, a wall, etc.) having one or more lines and / or faces in contact with the plane of the floor surface (ground). The gravity horizontal object can also have a horizontal to vertical ratio (a value obtained by dividing the horizontal length by the vertical length) of 1 or less.

[0143] On the other hand, a second embodiment of the floor (ground) plane detection using multiple objects (S3420) starts with a step (S3422b) of detecting multiple similar objects whose mutual similarity is equal to or greater than a predetermined value. For example, doors, desks, chairs, etc. of a certain design can be such similar objects.

[0144] Considering that the video (image) processing method of the present disclosure can be performed continuously and repeatedly, if the video (image) processing method has already been provided in the space in which the similar object exists, in step (S3422b), it is possible to detect the similar object by referring to the position and length of the similar object previously detected.

[0145] Following step S3422b, the second embodiment of the process S3420 further includes a step S3424b of detecting at least one bottom object, which is an object located at the bottom of the multiple similar objects. As in the example shown in Fig. 7a, the chair leg and the door are bottom objects because the bottom of the chair leg and the door contact the floor surface (ground).

[0146] Following step (S3424b), the second embodiment of process (S3420) further includes a step (S3426b) of detecting floor (ground) contact points which are contact points of each of the bottom-most objects with the bottommost level of the object segmentation, and further includes a step (S3428b) following step (S3426b) of connecting the floor (ground) contact points to generate a three-dimensional reference plane and extending the three-dimensional reference plane to generate a floor (ground) plane.

[0147] As an example of a modification of the second embodiment described above, the process (S3420) includes a step (S3422b') of detecting a number of similar objects whose mutual similarity is equal to or greater than a predetermined value, a step (S3424b') of determining whether one of the number of similar objects is located completely above another of the number of similar objects, and a step (S3424b') of classifying objects having different floor surfaces (ground surfaces) among the number of similar objects based on whether they are located completely above each other. The method may include a step of including (S3426b') and a step of generating (S3428b') a plane of the layer floor surface (ground) (ground) as a plane of two or more floor surfaces (ground) by connecting floor surface (ground) contact points, which are contact points of the layer floor surface (ground) set objects, which are objects belonging to each of the plurality of different layer floor surface (ground) sets, to the lowest level of the object segmentation of the layer floor surface (ground) set objects, to generate a three-dimensional reference plane and extending the three-dimensional reference plane to generate a plane of the layer floor surface (ground) (ground).

[0148] FIG. 7b is an exemplary diagram for explaining multiple steps of detecting two or more floor (ground) planes in an image processing method according to an embodiment of the present disclosure.

[0149] Step (S3422b') is similar to step (S3422b), so referring to FIG. 7b, the "full upper stage" of steps (S3424b') and (S3426b') will be explained. First, it is possible to assume that objects whose similarity is equal to or greater than a predetermined threshold have the same height h within the error range.

[0150] When objects are located apart from one another along the vertical axis (y-axis) in the overall image, it is possible to detect (i) that there is a distance between the objects, or (ii) that there is a difference in the height of the floor surface (ground) supporting the objects. If the bottom left corner of the overall image is taken as the origin of the coordinate system, when there is a distance between the objects, an object (700b) having a lower y-axis coordinate value compared to another object (700a) has a higher height (h b ) is larger and has a relatively higher y-axis coordinate value. a ) is the height (h b ) must be smaller than the

[0151] In other words, objects whose heights change at a constant rate depending on the y-axis coordinate value (e.g., 700b and 700a), and objects whose horizontal (x-axis) coordinate values ​​differ but whose y-axis coordinate values ​​differ only by a certain level or less (e.g., 700b and 700d) are determined to be on the same bed surface (ground), and it is possible to generate the corresponding set of bed surfaces (ground) mentioned above.

[0152] Suppose that the y-axis coordinate value is larger than the other object (700a), but the height (h a ) at a constant rate to the height (h c ) does not decrease, the height (h cIf there is an object (700c) with the same or higher floor surface (ground surface), it can be determined that it exists on a different floor surface (ground surface), and therefore it is possible to generate a set of different floor surfaces (ground surfaces).

[0153] By repeating such inference assumptions, it is possible to generate the above-mentioned set of bedding surfaces (ground surfaces).

[0154] Alternatively, instead of the height of each object, the height or area of ​​a three-dimensional reference plane that each object constitutes can be used.

[0155] On the other hand, in detecting the floor (ground) plane (S3420), the third embodiment using Manhattan space begins with a step (S3422c) of detecting at least one length-equivalent object that belongs to a length-equivalent category, which is a category that satisfies the condition that the deviation in length of at least a portion is smaller than a predetermined standard, from among objects or partial objects of such objects that belong to a particular category among at least one category.

[0156] Such length-equivalent objects may include desks, which have a relatively small deviation in height.

[0157] Following step (S3422c), the third embodiment of the process (S3420) further includes a step (S3424c) of detecting a Manhattan space generated by the collection of length-equivalent objects, a step (S3426c) of detecting a floor (ground) of the Manhattan space, and a step (S3428c) of horizontally extending the floor (ground) of the Manhattan space to generate a floor (ground) plane.

[0158] More specifically, the detection of Manhattan space (S3424c) can be composed of a first step (S3424c-1) of detecting floor (ground) objects, which are objects that constitute the floor (ground) surface, among the objects, and generating a boundary of the floor (ground) object, a second step (S3424c-2) of detecting wall objects, which are objects perpendicular to the boundary of the floor (ground) object, and a third step (S3424c-3) of detecting ceiling objects, which are objects perpendicular to the wall objects, as objects other than the floor (ground) objects.

[0159] That is, the Manhattan space here refers to a space surrounded by a floor (ground) object, a wall object, and a ceiling object. Examples of wall objects perpendicular to the floor (ground) object include glass windows and doors.

[0160] Unlike the previous embodiments, the fourth embodiment, which uses multiple objects having the same pattern in detecting the floor (ground) plane (S3420), begins with a step (S3422d) of detecting multiple identical pattern objects, which are multiple objects having the same pattern.

[0161] Following step (S3422d), the fourth embodiment of the process (S3420) further includes a step (S3424d) of detecting the lower part of the same pattern object if the same pattern object is detected, and a step (S3426d) of measuring the relative distance between the same pattern objects based on one of an occlusion between the same pattern objects and a difference in length in the image between the same pattern objects.

[0162] For example, in step (S3426d), if one object obscures the object segmentation of another object, the one object can be detected as being closer to the system than the other objects, and if one object is smaller than the other objects among multiple objects having the same pattern, the one object can be detected as being farther from the system than the other objects.

[0163] Following step S3426d, the fourth embodiment of the process S3420 further includes step S3428d of generating a virtual plane from floor (ground) contact points of two or more of the same pattern objects, or from three or more points included in one of the same pattern objects. When there are a plurality of generated virtual planes, a skilled technician can easily understand that it is possible to generate layered virtual planes, each layer being made up of the plurality of virtual planes, based on differences in length, state, and position.

[0164] Following step (S3428d), the fourth embodiment of the process (S3420) further includes a step (S3429d) of expanding the virtual plane to generate a floor (ground) plane.

[0165] Returning to the 3D measurement part in step (S3400) and continuing to explain this, the 3D measurement can further include, after detecting the floor (ground) plane (S3420), a process (S3430) of setting a virtual length reference line on the detected floor (ground) plane; and a process (S3440) of measuring the distance between the object and the system position, which is the origin of at least one of the orientations and coordinates of the spatial object, or the position of the object based on the system position.

[0166] On the other hand, the two-dimensional measurement in step (S3400) may include orientation and orientation measurement to calculate the two-dimensional orientation and two-dimensional orientation of the object based on the relative positions between multiple partial objects of the object, and area measurement to calculate the two-dimensional area of ​​the object, which can be performed by the detailed classification module (4300) using the measurement model (M431) of the detailed classification model (M430).

[0167] Due to the nature of the 2D measurement, area here means the area in the whole image or object image without considering the depth. This 2D area measurement can be done by object segmentation classification and measurement.

[0168] In addition, in step (S3400), in order to generate characteristics of the object, the detailed classification model (M430) may further include a deepened characteristic model (M432) for calculating at least one of information of partial objects which are part of the object or which are components belonging to the object, deepened classification information of the object, the main color of the object, the subordinate of the object, the subject of the object, one or more materials of the object, the transparency of the object, and whether the object can move under its own power.

[0169] Here, the deepened classification information of the object refers to information obtained by dividing the category of the object by a deepening division. For example, if the category of an object is a dog, the deepened classification information may be the breed of the dog.

[0170] Attitudes and behavior In addition, the detailed classification model (M430) may further include a posture discrimination model (M433) that calculates the posture of the object based on the processing results of the measurement model (M431) and the deepened characteristic model (M432), and may further include an action discrimination model (M434) that classifies the actions of the object based on the temporally consecutive postures calculated from the posture discrimination model (M433).

[0171] For example, the processor 120 of the computing device 100 may use the pose determination model M433 to obtain pose information associated with an object from analysis information corresponding to the object.

[0172] Additionally, the processor (120) of the computing device (100) can use the action discrimination model (M434) to obtain action information related to the object from the posture information related to the object.

[0173] Here, the concept of "action" includes both behavior that does not reflect the context and behavior that reflects the context, and behavior will be described later.

[0174] Since the posture determination method may differ depending on the category of the object, the posture determination model (M433) can be a posture determination model by category that differs depending on the category. For example, a dog posture determination model that calculates the posture of a dog when the dog is sitting and a human posture determination model that calculates the posture of a person when the person is sitting can be different. Therefore, in the posture determination model (M433), it is possible to determine a posture determination method to be applied to an object from among a plurality of posture determination methods based on analysis information corresponding to the object. The posture determination model (M433) can apply different posture determination methods depending on classification information indicating the category of the object.

[0175] Similarly, since the method of identifying an action may differ depending on the category of the object, the action identification model (M434) can be a category-specific action identification model that differs depending on the category. The action identification model (M434) can provide different action identification methods depending on the classification information indicating the category of the object.

[0176] The posture determination model (M433) can generate posture information for an object from the analysis information, including information on an N-dimensional (e.g., one-dimensional, two-dimensional, three-dimensional, etc.) posture corresponding to the object. The posture information for an object can include information on temporally consecutive postures of the object.

[0177] The action discrimination model (M434) can obtain action information of the object including an N-dimensional (e.g., one-dimensional, two-dimensional, three-dimensional, etc.) movement vector corresponding to the object from the posture information of the object. The action discrimination model (M434) can also generate action information of the object related to the context of the object including an N-dimensional (e.g., one-dimensional, two-dimensional, three-dimensional, etc.) action classification corresponding to the object from the posture information of the object. The context of the object can include at least one of the state of the object and / or the purpose of the action. The action classification can be calculated by identifying the position of a partial object included in the object, determining the posture of the object based on the position of the partial object, and determining the action classification of the object based on the posture of the object. The partial object can include at least one of a part of the object or a component belonging to the object.

[0178] The posture and action determination can be performed two-dimensionally or three-dimensionally. Thus, the movement vector can include a two-dimensional movement vector including at least one of the two-dimensional direction and / or velocity of the object. The movement vector can also include a three-dimensional movement vector of the object calculated based on at least one of the position, velocity and / or acceleration of the image capture device that captured the video (image).

[0179] In an embodiment in which posture and action discrimination is performed two-dimensionally, in step (S3400), the detailed classification module (4300) can also calculate two-dimensional movement information of the object, including a two-dimensional movement vector of the object and a two-dimensional action classification of the object, from a plurality of temporally consecutive object images (images). Therefore, the two-dimensional action classification can be calculated by performing a step (S3450a) of identifying the positions of each of the partial objects contained in the object, a step (S3460a) of determining a two-dimensional posture of the object based on the relative positions of each of the partial objects using a posture discrimination model (M433), and a step (S3470a) of determining a two-dimensional action classification of the object based on the temporally consecutive two-dimensional postures of the object using an action discrimination model (M434).

[0180] Here, the two-dimensional movement vector indicates the two-dimensional direction and speed of the object, and the two-dimensional action classification indicates the type of action determined from the two-dimensional posture of the object.

[0181] On the other hand, in an embodiment in which posture and action discrimination is performed three-dimensionally, in step (S3400), the detailed classification module (4300) can also calculate three-dimensional movement information of the object, including a three-dimensional movement vector of the object and a three-dimensional action classification of the object, from a plurality of temporally consecutive object images (images). Therefore, the three-dimensional action classification can be calculated by executing a step (S3450b) of identifying the positions of each of the partial objects contained in the object, a step (S3460b) of determining the three-dimensional posture of the object based on the relative positions of each of the partial objects using a posture discrimination model (M433), and a step (S3470b) of determining the three-dimensional action classification of the object based on the temporally consecutive three-dimensional postures of the object using an action discrimination model (M434).

[0182] Unlike 2D posture and action determination, 3D posture and action determination also needs to reflect the movement in the depth direction from the image capture device, so the computing device (100) can use the associated sensor (340) to calculate or estimate at least one (one or more) of the position, velocity and / or acceleration of the image capture device and reflect this in calculating the 3D movement vector of the object.

[0183] For example, when the system of the present disclosure drives the gimbal motor (330) to rotate the gimbal axis and track an object, such as a bicycle, traveling in a rightward direction based on the system, and calculates the three-dimensional movement vector of the bicycle, the three-dimensional movement vector can reflect both the movement vector caused by the motor (330) and the movement vector in the video (image).

[0184] To calculate the three-dimensional position and three-dimensional movement vector of the object, the standard focal length of the camera (200) and / or the standard focal length of the whole image (picture) or the object image (picture) can be used. The relationship between the distance between the camera (200) and the object, the standard focal length, the three-dimensional height (actual height) of the object, the height of the image (i.e., the vertical size of the image), the height of the object in the image (picture), and the height of the camera (200) is given by the following equation 1.

[0185]

number

[0186] Using Equation 1, it is possible to calculate the distance from the image capture device (200) to an object or the three-dimensional height of the object.

[0187] The distance reference line, which is one of the references used to measure the distance from the image capture device (200) to an object, can be, for example, a distance reference line set using concentric spheres based on the system of the present disclosure when two or more objects with the same length are detected. This line is composed of a reference plane circle and a measurement plane circle, which will be described later. The steps for setting this line are as follows:

[0188] FIG. 7c is a conceptual diagram for explaining a method for generating and using a reference plane circle and a measurement plane circle in a video (image) processing method based on an embodiment of the present disclosure.

[0189] According to FIG. 7c, first, a reference plane circle (750) is generated along a plane direction perpendicular to gravity, where at least one (one or more) of an accelerometer, a gyroscope and / or the above-mentioned gravity horizontal object can be used when determining whether it is perpendicular to gravity.

[0190] Here, the reference plane circle (750) corresponds to the above system, that is, the surface of the concentric sphere (740) with the imaging device (200) as the origin, that is, the cross-sectional shape of the concentric sphere cut in the xy plane.

[0191] Next, from among the equal-length objects, equal-length objects, which are objects whose lengths are the same as each other, are detected.

[0192] If the collection of multiple points on the concentric spheres in Figure 7c that are separated by an angle (θ) from the z-axis is referred to as a measurement plane circle, the difference in the angle (θ) coordinate values ​​between the above objects is measured by utilizing the fact that the above objects of the same length are interposed between the measurement plane circles for each angle (θ).

[0193] In this way, the difference in the angular coordinate values ​​can be used to measure the distance from the image capture device (200) to each object, and at least one (or more) optical characteristics of the standard focal length, lens and / or aperture size of the image capture device can be used as an auxiliary measure.

[0194] Meanwhile, the models mentioned in this disclosure, including the action discrimination model (M434), can be generated by supervised learning or reinforcement learning. It is well known that the analysis values ​​(text or classification index) and video information corresponding to each of the actions of the object can be used as training data for supervised learning. In addition, reinforcement learning can be used in which action information based on the action discrimination model (M434) is output to the user, and when the user views it and gives positive or negative feedback, the model is corrected using the feedback.

[0195] context The detailed classification model (M430) as a set may further include a context model (M435) that infers a context from the entire video (image). A context describes the state of an object and the purpose of an action. As an example, when a video (image) of a person peeling an orange with a knife in a kitchen is input, the context model (M435) may output the sentence "A person in the kitchen is peeling an orange with a knife" or a signal corresponding thereto as a context.

[0196] A behavior refers to an action of an object that does not reflect the context. For example, a person running on a basketball court and a person running on a treadmill are considered to be in the action of "running." The behaviors described above can be synthesized with the context described below to determine an action that takes the context into account. According to this, the former corresponds to the action of "playing basketball" and the latter corresponds to the action of "using a treadmill."

[0197] Specifically, the context may be an object context that includes at least one of the actions and states of each of the individual objects appearing in the overall image, and context-interacting objects, which are other objects that are detected as interacting with the individual objects through the actions.

[0198] Furthermore, the context may be a spatial context that includes a location, which is the type of the spatial object, at least one of the actions and states of each of the individual objects, an actor, which is an individual object corresponding to the subject of the action, and a context interacting object, which is an object detected as interacting with the individual object through the action, inferred from each of the spatial objects and individual objects other than the spatial objects that appear in the overall image (image).

[0199] In one embodiment, the synthesis of actions and context to determine a context-aware action can be implemented by supervised learning using an artificial neural network model. For example, an artificial neural network model can be trained using training data in which a sequence of images, i.e., a video scene, is used as input data, and linguistic analysis (i.e., data expressed in language) including the actions and spatial context of objects is labeled with ground truth output data.

[0200] This is as described in the non-patent literature paper Wu, Z., Yao, T., Fu, Y., & Jiang, Y.-G. (2017). Deep learning for video classification and captioning. In Frontiers of Multimedia Research (pp. 3-29). ACM.

[0201] On the other hand, for a more specific method of determining the behavior of each object, see Wu, C.-Y., Girshick, R., He, K., Feichtenhofer, C., & Krahenbuhl, P. (2020). A Multigrid Method for Efficiently Training Video Models. In 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. https: / / doi.org / 10.1109 / cvpr42600.2020.00023.

[0202] On the other hand, there are cases where an action is heterogeneous in the spatial context, and in this case, there is a possibility that the action may be analyzed in a way that does not match the context. For example, an object that is a person running in a cafe can be analyzed as "busy moving" rather than as "running". Such inconsistencies occur because the action does not adequately reflect the context, or the context does not adequately reflect the action. That is, since actions can be inferred from contexts and contexts can also be inferred from actions, a cyclical process may be required in which contexts are inferred from actions and actions are inferred from contexts again. This can be achieved, for example, by using a neural network model such as a recurrent neural network (RNN) or a bidirectional LSTM as disclosed in the non-patent literature article Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, SW (2018). Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features. IEEE Access, 6, 1155-1166. https: / / doi.org / 10.1109 / access.2017.2778011.

[0203] Priority The following is a detailed explanation of the priority order of objects, which is closely related to the importance of the objects mentioned above. Here, the importance is a value used to assign a priority order, but usually, after calculating the importance of each object, it is possible to set the priority order based on the importance.

[0204] The object priority can include a privilege-based priority that is assigned to each of the objects based on the privilege of the object, which assigns a priority according to the privilege of each of the classified objects, for example, in the system of the present disclosure, if an object that is a particular user has the highest privilege, the particular user can be given a high priority.

[0205] The authority-based priority can be calculated by performing a step of determining whether the object is a predetermined authority that can use the computing device based on the characteristics of the object analyzed using a detailed classification model (M430), and if the object is a predetermined authority, setting the authority-based priority of the object to a predetermined ranking for that authority.

[0206] For example, since it is expected that the authority will be a person, it is possible to first detect objects that are people and set priority based on authority only for the detected human objects.

[0207] Additionally, the computing device of this disclosure can use the above-described deepened classification information to identify an individual person from among a plurality of people, and can use object characteristics, such as the color of the object, or characteristics of a partial object, or object record of the object, to identify the individual person.

[0208] On the other hand, the priority of objects may be a classification-based priority that is assigned to each of a plurality of object sets that are divided by at least one characteristic of the objects, including at least the category of the object or deeper classification information of the object.

[0209] The classification-based priority may be a priority given to categories or in-depth classification information such as people, animals, and other objects as set in advance, or may be manually set by an authority of the system. As another example, the computing device of the present disclosure may automatically set the priority of each category or in-depth classification information variably based on the pattern of use of the system of the present disclosure by an authority of the system.

[0210] Next, the priority of objects may include a size-based priority assigned to each of a plurality of object sets separated by at least one characteristic of the object, including at least the size of the object.

[0211] The size-based priority can be calculated by the detailed classification module (4300) performing the steps of obtaining object segmentation of the object, selecting from the objects those objects whose object segmentation size occupies a proportion of the entire video (image) that is equal to or greater than a predetermined proportion, and setting the size-based priority according to the size of the object segmentation of the selected objects.This is a method in which the larger the object is or the closer the distance between the object and the system of this disclosure, the larger the proportion of the object segmentation of the object in the entire video (image), and the higher the proportion, the more important the object is determined to be, and a higher priority is set.

[0212] In addition, the priority of objects may include an action-based priority that is assigned to each of a plurality of object sets that are divided by at least one state of the object, the state including at least the action of the object.

[0213] Specifically, the detailed classification module (4300) can search for objects that perform a particular action and assign a priority corresponding to that action.

[0214] For example, if a person falling is detected, the highest priority can be set to the person who fell. As another example, if there is a human object that issues commands by a specific gesture, such as swiping in the air, the human object that issues commands can be given a high priority.

[0215] As a final example, object priority may include a context-based priority assigned to each of a plurality of object sets separated by at least one of the object states, the context including at least the object's context.

[0216] According to the context-based priority, the detailed classification module (4300) can infer object context and spatial context from the overall image (image) by performing state or spatial analysis on the overall image (image), and set a relatively high priority to the actor or the context interacting object based on whether the object is a context interacting object, which is an object that is detected as interacting with the actor or the actor due to the action of the actor according to at least one of the inferred object contexts and spatial contexts.

[0217] For example, if the spatial context is "people of child age playing baseball on a field," a relatively high priority can be given to a person detected as holding a baseball bat.

[0218] The steps and processes described in this disclosure do not necessarily have to be performed in the order described, unless logically inconsistent or otherwise defined by the context, and it will be obvious to one of ordinary skill in the art that each of the steps and processes can be performed simultaneously or without simultaneousness.

[0219] Additionally, the above-mentioned steps may be performed once, but preferably, as described above, the above-mentioned steps can be performed in real-time and / or iteratively to obtain time-continuous images.

[0220] That is, the image processing method of the present disclosure may further include a step (S4000) of returning to the image acquisition step (S1000) to acquire a new entire image, and at this time, the image acquisition step (S1000) can be executed again by controlling available resources by the tracking controller (4400).

[0221] Here, the tracking controller (4400) is a module similar to the video (image) input module (4100), object analysis module (4200) and detailed classification module (4300), and therefore performs the function of enabling the system of the present disclosure to move the camera (200) toward a target object or target space, i.e., to implement tracking, where available resources refer to the hardware and / or software that enables such tracking.

[0222] In one embodiment, the tracking controller (4400) may include a target position determination module (4420) that determines a target position and magnification corresponding to the composition to be captured by the imaging device (200), and a tracking resource controller (4440) that controls the available resources, i.e., hardware and software resources, to obtain the entire image (image) of the target position.

[0223] Target Position Determination Module 8a-8c are flow charts showing, by way of example, several techniques that may be used to determine a target position in the image processing method of the present disclosure.

[0224] In one embodiment shown using FIG. 8a, the target position determination module (4420) is capable of performing a step (S4110a) of obtaining an object priority of at least one candidate object appearing in the overall image (image), a step (S4120a) of obtaining an object segmentation of the candidate object, a step (S4130a) of identifying a context interacting object which is an object detected as interacting with the candidate object along the inferred context in the overall image (image), a step (S4140a) of determining the composition captured by the imaging device (200) to include at least one target object and the context interacting object of the target object which is a candidate object having an object priority of a predetermined rank or higher, and an additional composition prediction step (S4150a) of determining the target position and magnification by extending the composition along the direction of the target object.

[0225] The identification of contextually interacting objects can be performed by a context model (M435). The context model (M435) can be generated by supervised learning, as disclosed in the non-patent literature paper Shin, D., & Kim, I. (2018). Deep Image Understanding Using Multilayered Contexts. Mathematical Problems in Engineering, 2018, 1-11.

[0226] More specifically, the additional composition prediction step (S4150a) may include a step of acquiring a direction of the target object (S4152a) and a step of acquiring a velocity of the target object (S4154a). As described above, the direction of the target object may be acquired based on relative positions between a plurality of partial objects included in the target object, and the velocity of the target object, i.e., a three-dimensional motion vector, may be calculated based on a two-dimensional motion vector acquired from an image (image) and information from the sensor (340).

[0227] Next, the additional composition prediction step (S4150a) may further include a step (S4156a) of calculating the range of influence of the target object, which is the range in which interaction with the target object can be performed, based on the direction and speed of the target object, and a step (S4158a) of determining the target position and magnification by expanding the composition to reflect the range of influence.

[0228] Here, the influence range refers to the range of space that the contextually interacting object may occupy or interact with at least temporarily within a specified time range starting from the present, i.e., the range of space where physical contact by the target object is possible, for example, the range where a part of a person's body can physically come into contact, or the range in which a contextually interacting object may exist where the target object can send and receive signals through the eyes, ears, mouth, etc., for example, the field of view (FOV) where signals can be sent and received from the target object.

[0229] As an example, in step S4156a, the range of influence of the eye may be 20 m, and the range of influence of the hand may be 1 m. This may be combined with the action information (action classification) of each object and used when adjusting the composition. For example, if a target object is a "child standing at bat with a bat" in hand, in step S4158a, the composition may be adjusted to include the direction of the child's gaze and the object of the bat held by the child.

[0230] On the other hand, in another embodiment shown using FIG. 8b, the target position determination module (4420) can determine the composition to be captured based on the priority of the object; specifically, the target position determination module (4420) can execute a step (S4120b) of obtaining an object priority of at least one candidate object appearing in the entire image (picture), a step (S4140b) of determining at least one target object, which is a candidate object having an object priority of a predetermined rank or higher, and a step (S4160b) of determining the composition to be captured by the photographing device (200) so as to include the target object.

[0231] Referring to FIG. 8c, in yet another embodiment reflecting the priority of objects, the target position determination module (4420) performs the steps of: acquiring an object priority of at least one candidate object captured in the overall image (image) (S4120c); generating a virtual segmentation including the candidate object according to the object priority of the candidate object; the virtual segmentation is generated such that the area of ​​the virtual segmentation increases as the object priority increases (S4140c); and multiplying the area of ​​the virtual segmentation by a predetermined increasing function (S4140c). It is possible to execute a step (S4160c) of calculating an attractive force between the virtual segmentations, which monotonically increases in response to the distance between the candidate objects, a first repulsive force between the candidate objects, which monotonically decreases in response to the distance between the candidate objects by a predetermined first decrease function, and a second repulsive force between the candidate object and the frame boundary line of the entire image (picture) by a predetermined second decrease function, and determining a balance state of the attractive force, the first repulsive force and the second repulsive force, as well as a step (S4180c) of determining a target position and magnification corresponding to the composition by calculating a target center, which is the central point of the at least one candidate object in the balance state.

[0232] Tracking Resource Controller The tracking resource controller (4440) may include an image (picture) conversion controller (4442) that controls the acquisition of a new image (picture) from the image (picture) acquired from the camera (200) without controlling the attitude of the gimbal (300) and the camera (200), and a frame conversion controller (4444) that controls the gimbal (300) and the camera (200) to acquire the new image (picture) when the computing device (100) is linked to a gimbal (300) to which the camera (200) is attached and which has one or more rotation axes that control the attitude of the camera (200).

[0233] The frame conversion controller (4444) and the image conversion controller (4442) can operate to complement each other, and the tracking resource controller (4440) can first obtain a new image (image) of the desired resolution from the frame conversion controller (4444), and if that acquisition fails, can control the acquisition of a new image (image) through image reconstruction by the image conversion controller (4442).

[0234] In a specific embodiment, the image conversion controller (4442) can execute a step (S4310) of determining whether there are idle resources available for image reconstruction, a step (S4320) of loading the entire image or a portion of the entire image into memory as an original image if the idle resources are available, and a step (S4330) of cropping the original image according to the target position and magnification, or performing super-resolution if the resolution of the original image does not reach a predetermined threshold, to obtain the new image that matches the target position and magnification.

[0235] It is well known to those of ordinary skill in the art that super-resolution can be performed using neural networks, such as, for example, autoencoders.

[0236] Meanwhile, the frame conversion controller (4444) may include a gimbal controller (4444a) that controls the direction of the imaging device (200) by operating one or more rotation axes of the gimbal (300) to achieve the target position (i.e., to reach the target position), a zoom controller (4444b) that controls the zoom-in and zoom-out of the imaging device (200) to reach the magnification, and a front and rear imaging device controller (4444c) that controls the available resources to capture the environment while reducing hardware operation, and among them, it includes at least the gimbal controller (4444a).

[0237] The image capturing device (200) may be configured in two or more units, for example, image capturing devices (200) mounted on the front and rear of the mobile terminal (100). The front and rear image capturing device controller (4444c) can scan the entire space surrounding the system, that is, the surrounding space, using the front image capturing device (200a; not shown) and the rear image capturing device (200b; not shown), and can select the image capturing device (200) to be used from the front and rear image capturing devices based on the user's convenience.

[0238] For example, the front and rear camera controller (4444c) can select the front camera (200a) as the camera to be used to capture an image (picture) of the user when the user is looking at the display of the portable terminal (100). In yet another example, the front and rear camera controller (4444c) can control the recognition of an object as a front camera (200a), and then, if a video (image) of the object with a higher resolution than that of the front camera (200a) is required, to recognize the object as a rear camera (200b) having the higher resolution.

[0239] In addition, the front and rear camera controller (4444c) can use the front camera (200a) and the rear camera (200b) simultaneously or sequentially to capture images of the surrounding space while minimizing rotation of the gimbal (300) axis.

[0240] Meanwhile, the frame conversion controller 4444 can control the orientation of the photographing device 200 to either portrait or landscape depending on the ratio of width to height (a value obtained by dividing the width by the height) of at least one specific object or a specific partial object included in the specific object so that the composition includes the entire object or the specific partial object. For example, this can be performed by rotating the roll axis R of the gimbal 300. In the ratio of width to height, width can mean perpendicular to the gravity vector. In the ratio of width to height, height can mean horizontal to the gravity vector. Thus, the ratio of width to height can be calculated by taking a part of a reference plane as the width. The reference plane can be a plane perpendicular to the gravity vector and / or a plane in contact with the floor (ground). However, the reference plane is not limited thereto, and various planes can be set as the reference plane. It is also possible that the width to height ratio does not change trend (eg, greater than or less than 1) depending on the object's orientation.

[0241] Specifically, the frame conversion controller (4444) can control the orientation of the imaging device to either a vertical or horizontal direction so that the object segmentation of the at least one specific object or a specific partial object, or an object box containing the at least one specific object or a specific partial object, does not contact the frame boundary of the entire image.

[0242] 9a and 9b are flow charts showing, by way of example, several methods that may be used to control the orientation of the image capture device in the image processing method of the present disclosure.

[0243] Referring to FIG. 9a, an embodiment of the frame conversion controller (4444) for controlling the orientation of the photographing device will be described in more detail. When an object segmentation or object box of the at least one specific object or a specific partial object makes a first contact with a frame boundary of the entire image (picture), the frame conversion controller (4444) controls the gimbal to move in the opposite direction to the first contact position within a range in which the center point of the object segmentation or the object box exists within the composition, thereby converting the first contact into a control signal. and if, after the movement, the object segmentation or the object box makes a second contact with the frame boundary opposite the position of the first contact, then a step of resolving the second contact (S4220a) can be performed through at least one of the following operations: (i) switching the orientation of the camera from portrait to landscape, and (ii) controlling zoom-in and zoom-out of the camera.

[0244] Referring to FIG. 9b, another embodiment of the frame conversion controller 4444 will be described in detail. In the step of acquiring a ratio characteristic among the characteristics of the at least one specific object or specific partial object, the frame conversion controller 4444 may acquire a ratio characteristic that is predetermined for each category of the specific object or the specific partial object, or may acquire a ratio characteristic that is predetermined for each detailed classification of the specific object or the specific partial object, which is a ratio of the width to the height (a value obtained by dividing the width by the height) calculated for each category by measuring the specific object or the specific partial object. It is possible to execute a step (S4210b) including a detailed classification ratio characteristic, which is the ratio of width to height (the value obtained by dividing the width by the height) calculated for each detailed classification by measuring the specific object or a specific partial object, and the ratio of width to height (the value obtained by dividing the width by the height) of the composition determined by the target position determination module, as well as a step (S4220b) of determining whether or not to perform a roll rotation, which is an operation of switching the orientation of the photographing device from one of portrait and landscape to the other, based on the ratio characteristic, and adjusting the orientation of the photographing device depending on whether or not to perform the roll rotation.

[0245] For example, the width-to-height ratio of a television (width divided by height) is greater than 1, so the orientation of the imaging device (200) can be adjusted horizontally, and the width-to-height ratio of a standing person (width divided by height) is less than 1, so the orientation of the imaging device (200) can be adjusted vertically.

[0246] After step S4220b, the frame conversion controller 4444 of this embodiment may additionally execute step S4230b of, when the object segmentation or object box of the at least one specific object or the specific partial object contacts the frame boundary of the entire image, releasing the contact through at least one of the following operations: (i) switching the orientation of the photographing device from one of portrait and landscape to the other; and (ii) controlling zoom-in and zoom-out of the photographing device. In this case, if the object segmentation or object box does not contact the frame boundary, the process ends at step S4220b, and step S4230b is not executed.

[0247] OCR (Optical Character Recognition) Returning to the detailed classification model (M430), the OCR for the detailed classification model (M430) will be described below. The classification model (M430) may further include an OCR model (M436) that reads text displayed on the surface of the object. Thus, the processor (120) of the computing device (100) may use the OCR model (M436) to obtain characters contained in the object from the analysis information corresponding to the object.

[0248] Correspondingly, the aforementioned step (S3400) may include a step (S3500) of calculating the text of the object as part of the object's characteristics by the fine-grained classification module (4300) performing OCR using the OCR model (M436) of the fine-grained classification model (M430).

[0249] To explain step S3500 in more detail, the OCR model M436 may first include a step S3520 of determining whether or not characters are displayed on the surface of the object.

[0250] In step S3520, it is known that by detecting characters using deep learning or conventional OCR technology, it is possible to determine the object in which the characters are displayed.

[0251] Following step (S3520), the OCR model (M436) can execute an OCR execution step (S3540) in which, if characters are displayed on the surface of the object, the OCR model (M436) performs OCR on the displayed characters (e.g., the entire character and / or a character set) and inputs (saves) the result text, which is the result of the OCR, as a property of the object.

[0252] According to one embodiment of the present disclosure, the OCR model (M436) can perform a step of determining whether or not characters are displayed on a surface of an object, and if it is determined that characters are displayed on the surface of the object, performing OCR on the entire character including the characters.

[0253] According to another embodiment of the present disclosure, the OCR model (M436) may perform a step of determining whether characters are displayed on a surface of an object, and if it is determined that characters are displayed on the surface of the object, performing OCR for each character set.

[0254] How to classify character sets At least one of the processor (120) or the OCR model (M436) can classify a character into a character set based on at least one of the character type, shape, size, or arrangement. The processor (120) can also classify a character set that is highly likely to have high relevance in analyzing a certain context.

[0255] The method of classifying the character set to which a character belongs or the character set to which a context belongs may be as follows: For example, at least one of the processor (120) or the OCR model (M436) may classify the character set based on the characteristics of the character set.

[0256] The characteristics of the character set may include one or more of the following: the language type of the characters, line spacing, character spacing, character width (e.g., horizontal to vertical ratio), size, weight, color, font, style, characters associated with the beginning or end of a sentence, white space (e.g., white space with at least one boundary of the next sentence, object, or partial object), or the position where the character appears in the object. Thus, at least one of the processor (120) or the OCR model (M436) may classify the character set using one or more features of the language type of the characters, line spacing, character spacing, character width (e.g., horizontal to vertical ratio), size, weight, color, font, style, characters associated with the beginning or end of a sentence, white space (e.g., white space with at least one boundary of the next sentence, object, or partial object), or the position where the character appears in the object.

[0257] For example, at least one of the processor (120) or the OCR model (M436) may classify a complete sentence or a context according to the type of language the characters belong to. At least one of the processor (120) or the OCR model (M436) may classify a complete sentence or a context according to the type of language each character belongs to when the characters to be recognized are in multiple languages, such as the surface of a product instruction manual.

[0258] As another example, at least one of the processor (120) or the OCR model (M436) may segment paragraphs based on at least one of line spacing, character spacing, character width, or color when the sentences are of the same size and typeface.

[0259] As yet another example, at least one of the processor 120 or the OCR model M436 may segment the character set based on the top of a paragraph or where a paragraph begins, where the nature or title of the paragraph may be indicated using a bold style or different sized characters.

[0260] Character set objectification The processor 120 and / or the OCR model M436 may classify the character set based on the above-mentioned "character set classification method." The processor 120 and / or the OCR model M436 may recognize a character set object based on the character set. The character set object may be a character set set as one target. The character set object may include at least one of a character image, text information, or a characteristic of the character set. For example, the character set object may include a character image before OCR or natural language understanding (processing). As another example, the character set object may include text information after OCR or natural language understanding (processing). In this case, the character set object may be separated into multiple character set objects after OCR or natural language understanding (processing).

[0261] The processor (120) and / or the OCR model (M436) can classify one or more character sets displayed on the surface of an object or partial object. The processor (120) and / or the OCR model (M436) can also perform a contextual analysis based on at least one of information related to properties or a state of the object or partial object on which the character sets are displayed. Properties and / or states of the object and / or partial object can be included in the properties of the character sets.

[0262] The processor (120) and / or the OCR model (M436) can use the type and / or location of the object or partial object in which the character set is displayed as additional information for natural language understanding and / or context analysis.

[0263] For example, if the text displayed on the shirt (object or partial object) and the text displayed on the car (object or partial object) are identical, the processor 120 and / or the OCR model M436 may use the text displayed on either the shirt or the car (e.g., a brand name) as additional information. In this case, the processor 120 and / or the OCR model M436 may determine that the text displayed on the car reflects (is relatively likely to reflect) the characteristics of the object and / or the characteristics of the overall context, and may use the text displayed on the car as additional information.

[0264] When multiple characters are present in an object, the processor 120 and / or the OCR model M436 can obtain the characteristics of the object based on the positions at which the multiple characters are displayed. The multiple characters may have different meanings depending on their positions. For example, when characters displayed on the front of a shirt (object or partial object) and characters displayed on a label (e.g., a care label) of the shirt are simultaneously present, the processor 120 and / or the OCR model M436 can determine that the characters displayed on the label of the shirt reflect the characteristics of the object (determine that there is a relatively high probability that they do), and use the characters displayed on the label of the shirt as additional information.

[0265] The processor 120 and / or the OCR model M436 can perform OCR and / or natural language understanding (processing) for each character set object of the objectified character set. The processor 120 and / or the OCR model M436 can perform OCR and / or natural language understanding (processing) for one or more character set objects based on the characteristics of the character set. In this case, the processor 120 and / or the OCR model M436 can perform OCR and / or natural language understanding (processing) for one or more character set objects based on the characteristics of the character set, either sequentially or simultaneously.

[0266] The unit of OCR and / or natural language understanding (processing) is not limited to the entire character on the surface of an object or partial object, but the smallest unit can be a character or a character set classified according to the above-mentioned characteristics of the character set. Meanwhile, the OCR execution step (S3540) can be performed in various ways.

[0267] For example, the step (S3540) of performing OCR on a character (e.g., an entire character and / or a character set) performed by the processor (120) of the computing device (100) may include the steps of extracting at least one image (image) sample from an object in which the character is displayed, determining a boundary line of an area in which the character is displayed from the at least one image (image) sample, generating a character display image (e.g., an entire character display image (image) and / or a character set display image (image)) based on at least one of the boundary lines or boundary points that belong to the boundary line from the at least one image (image) sample, and performing OCR on the character display image (image).

[0268] The image sample may be an image pattern that is located at at least one of a boundary line and / or a boundary point. The image pattern may include at least one of a part of a character, a boundary of a character, a part of a character, and / or a background. The background may mean an image pattern that does not constitute a character.

[0269] The step of generating a character display video (image) based on at least one of the boundary lines or boundary points that are points belonging to the boundary lines from at least one video (image) sample, and the step of performing OCR on the character display video (image) may include the steps of obtaining a video (image) pattern located at at least one of the boundary lines or boundary points from at least one video (image) sample as a boundary marker, and generating the character display video (image) that is a video (image) including the boundary marker as a partial video (image) included in the video (image) of the object in which the character is displayed. The boundary marker may include at least one of a start marker and / or an end marker. The start marker may be a video (image) pattern corresponding to the first character of the characters. The end marker may be a video (image) pattern corresponding to the last character of the characters.

[0270] The step of acquiring an image (image) pattern located on at least one of the boundary lines or boundary points from at least one image (image) sample as a boundary marker may include a step of acquiring, using a tracking controller (4400), an image (image) of a character including a leading marker and having a character recognition rate equal to or higher than a threshold, and a step of determining whether the image (image) of the character includes an ending marker.

[0271] Then, the processor 120 can determine the image (image) of the text as the text display image (image) if the image (image) of the text includes an end marker.

[0272] If the character image does not include an end marker, the processor 120 can use the tracking controller 4400 to acquire an additional character image that includes a marker next to the last marker among the boundary markers included in the character image and has a character recognition rate equal to or higher than a threshold. The processor 120 can combine the additional character image with the character image to generate a combined character image. Then, if the combined character image includes an end marker, the processor 120 can determine the combined character image as the character display image.

[0273] Meanwhile, in another example, the step (S3540) of performing OCR on the character performed by the processor (120) of the computing device (100) may include the steps of: obtaining, using a tracking controller, an image of an object that includes the beginning of the character and has a character recognition rate above a threshold; performing OCR on an area of ​​the first sentence in the image of the object; and performing natural language understanding (NLU) on a first text, which is the primary result of the OCR, to calculate a first semantic value number, which is a numerical value of semantic value.

[0274] The processor (120) may determine the first text as a result text that is a result of the OCR if the first semantic value value is greater than or equal to a threshold value.

[0275] The processor (120) can perform OCR on the area of ​​the sentence next to the area of ​​the first sentence if the first semantic value value is less than the threshold value. The processor (120) can perform natural language understanding on the second text, which is the primary result of the OCR, to calculate a second semantic value value, which is a value of semantic value. Then, the processor (120) can determine the second text as a result text, which is the result of the OCR, if the second semantic value value is equal to or greater than the threshold value.

[0276] The area of ​​the first sentence may be the area occupied by the sentence that is detected as being placed first in a character placement scheme based on the language used in the sentence.

[0277] The computing device 100 may be in communication with an image capture device 200 and a gimbal 300. For example, the computing device 100 may include an image capture device 200 and be in communication with a gimbal 300 that controls the attitude of the image capture device 200.

[0278] The processor (120) of the computing device (100) can use the tracking controller (4400) to actuate one or more rotational axes of the gimbal (300) to control the orientation of the imaging device (200).

[0279] The processor 120 can use the tracking controller 4400 to capture images by zooming in and / or zooming out the imaging device 200.

[0280] 10a to 10c are flowcharts illustrating, by way of example, several methods used to perform OCR in the image processing method of the present disclosure, and FIGS. 11a to 11d are drawings shown as examples to explain several methods of performing OCR in the image processing method of the present disclosure.

[0281] First, referring to Figures 10a and 11a, in an embodiment using boundary markers, the OCR execution step (S3540a) begins with a step (S3542a) of extracting a video (image) sample from an object in which the character appears (e.g., a paper document of reference numeral 1110), and following step (S3542a) includes a step (S3544a) of defining from the video (image) sample a boundary line of the area in which the character appears (e.g., a closed curve of reference numeral 1130).

[0282] Here, the video (image) sample is a video (image) pattern that exists at the position of a boundary or boundary point, and the video (image) pattern can be composed of a part of a character, a boundary of a character, a part of a character, or a background. The above-mentioned boundary refers to the boundary of a group of characters shown on the surface of an object, and a boundary point refers to a set of multiple points that can compose a boundary. Here, the background refers to a video (image) pattern that does not itself compose a character.

[0283] In this embodiment, the OCR execution step (S3540a) further includes a boundary marker acquisition step (S3546a) following step (S3544a) of acquiring an image (image) pattern located on the boundary line or a boundary point, which is a point belonging to the boundary line, from the image (image) sample as a boundary marker. In this step, among the boundary markers, an image (image) pattern corresponding to the first character of the entire character can be referred to as a head marker, and among the boundary markers, an image (image) pattern corresponding to the last character of the entire character can be referred to as an end marker.

[0284] The boundary marker functions as a marker for determining the coordinate area of ​​the space occupied by each character belonging to the entire character in the object image (image).

[0285] A character boundary can be used as a boundary marker, i.e., one of the image patterns located on the boundary line or a boundary point that is a point belonging to the boundary line. A character boundary refers to a part of an image that constitutes a part of a character extending from a sentence along the horizontal or vertical direction of the sentence. For example, In the sentence "JPEG2025510794000003.jpg630", "JPEG2025510794000004.jpg66" JPEG2025510794000005.jpg45" and " "JPEG2025510794000006.jpg65" "JPEG2025510794000007.jpg63" can be said to be a boundary marker that is a part of a character and is the boundary of the character.

[0286] Also, a part (or the whole) of a letter can be used as one of the boundary markers. For example, in the sentence "ABCDEFG", A and G can be used as boundary markers that are parts of letters located at boundary points and / or borders.

[0287] The acquisition of the head marker or the end marker reflects the arrangement of characters based on the language of the sentence that the entire character is composed of. For example, in OCR for a book written in Korean and English, as shown in Fig. 11, the video (image) sample (1132) at the top left of the area where the characters are displayed can be the head marker, and the video (image) sample (1134) at the bottom right of the area can be the end marker.

[0288] More specifically, the boundary marker acquisition step (S3546a) may include a step (S3546a-1) of controlling available resources by the tracking controller (4400) to acquire a character image (image) that includes the above-mentioned leading marker and has a character recognition rate above a predetermined threshold.

[0289] Here, the character recognition rate refers to the rate at which a specific character is recognized as a specific text in a character video (image). For example, it is considered possible to estimate the presence of some characters in a long-distance video (image) showing an object on which characters are displayed, using deep learning or the like. However, it may be difficult to determine what character the character is due to the small size or low resolution of the long-distance video (image). In other words, the character recognition rate by OCR may be low.

[0290] In other words, a character image with a character recognition rate equal to or higher than a predetermined threshold indicates that the image has a resolution that allows OCR. In performing this step (S3546a-1), in order to obtain an enlarged character image with a sufficiently high resolution, the system of the present disclosure can control the zoom-in, zoom-out and rotation of the photographing device (200) through the tracking controller (4400). For example, referring to FIG. 11a, a rotation control (1160) and zoom-in control (1170, 1180) of the photographing device (200) are shown as examples, which allow the paper document (1110) and a person (1120) to be in the center of the image in a composition that includes both the paper document (1110) and the person (1120).

[0291] Following step (S3546a-1), this embodiment of the boundary marker acquisition step (S3546a) includes an end marker determination step (S3546a-2) of determining whether the end marker is included in the acquired character image (image), and a step (S3546a-3) of acquiring the character image (image) as the whole character display image (image) if the end marker is included in the acquired character image (image), and searching for a marker next to the last marker among the boundary markers displayed in the character image (image) if the end marker is not included in the acquired character image (image), and acquiring an additional character image (image) that includes the next marker and has a character recognition rate above a predetermined threshold, and combining the additional character image (image) with the character image (image), thereby re-executing from the end marker determination step (S3546a-2).

[0292] The control of available resources in step (S3546a-3) includes control over the gimbal (300), and for example, control of the gimbal (300) for a book written in Korean and English may be control that assists the imaging device (200) to scan from left to right and from top to bottom of the area in which the characters are displayed (e.g., rotation control of reference numeral 1190 in FIG. 11b).

[0293] Also, it is obvious to those skilled in the art that various image stitching methods can be used to stitch the images of the characters in step S3546a-3. Referring to Fig. 11b, image stitching can be performed using multiple overlapping image patterns (1136a, 1136b) in two or more images taken at a time or space interval.

[0294] Following the boundary marker acquisition step (S3546a) as described above, the OCR execution step (S3540a) of this embodiment can include a step (S3468a) of acquiring text by performing OCR on the entire character display image (image), which is an image (image) that includes both the beginning marker and the end marker as a partial image (image) that includes an image (image) of the object in which the character is displayed.

[0295] Meanwhile, referring to Figures 10b and 11c, in a second embodiment of the OCR execution step (S3540), the OCR execution step (S3540b) begins with a step (S3542b) of extracting an image (image) sample from the object in which the character is displayed, and following step (S3542b), includes a step (S3544b) of defining a boundary (1130) of the area in which the character is displayed from the image (image) sample.

[0296] Next, the OCR execution step (S3540b) includes a step of obtaining at least one image (image) pattern (1142a, 1142b) from the image (image) samples that is not located on the boundary line as a division marker, and performing OCR on the entire character display image (image), which is an image (image) generated by combining character images (images) using the division marker (S3546b).

[0297] In step (S3546b), the division marker refers to overlapping image patterns (1142a, 1142b) in two or more images taken at a time or space interval. By finding the same division marker in different images, it is possible to combine images using the division marker, that is, to join the images together.

[0298] Finally, referring to Figures 10c and 11d, in a third embodiment of the OCR execution step (S3540), the OCR execution step (S3540c) includes a step (S3542c) of controlling available resources by the tracking controller (4400) to obtain the object image (image) that includes the beginning of the entire character and has a character recognition rate above a predetermined threshold, i.e., so that the object image (image) has a predetermined resolution capable of performing OCR.

[0299] The policy of the tracking controller (4400) executing step (S3542c) can include not only the condition that the object image (image) has a certain resolution at which OCR can be performed, but also the condition that the object image (image) contains as much text as possible at such resolution.

[0300] Next, the OCR performing step (S3540c) further includes a step of performing OCR (S3544c) on the first sentence area (1152) in the acquired object image (image).

[0301] Here, the area of ​​the first sentence refers to the area occupied by the sentence that is detected as being placed first, taking into account the general arrangement of characters based on the language used in the sentence.

[0302] For example, when this step (S3544c) is performed for the first time, the first sentence area can be determined by scanning for delimiters that indicate sentence boundaries, such as ",", ".", and "?". When such a sentence code is scanned, the area from the first character to the delimiter can be considered as the first sentence area (1152).

[0303] As another example, when this step (S3544c) is executed for the first time, the area of ​​the first sentence (1152) can be determined as the area occupied by a sentence consisting of multiple characters following the first character, starting from the size of the first character and comparing the sizes of the next characters in sequence until the size of the characters changes by more than a predetermined threshold value.

[0304] As yet another example, when this step (S3544c) is executed for the first time, the area of ​​the first sentence (1152) can be determined as the area occupied by the sentence consisting of multiple characters following the first character, starting from the size of the character space between the first character and the next character, and comparing the size of the next character space in sequence until the size of the character space changes to or exceeds a predetermined threshold value.

[0305] Next, the OCR execution step (S3540c) further includes a semantic value numerical calculation step (S3546c) in which natural language understanding (NLU) is performed on the first text, which is the primary result of the OCR in step (S3544c), to calculate a semantic value numerical value, which is a numerical value of the semantic value of the first text.

[0306] Next, the OCR execution step (S3540c) further includes a step (S3548c) of completing the OCR by obtaining the first text as the result text if the semantic value value in step (S3546c) is greater than or equal to a predetermined threshold value, and performing OCR on the area of ​​the sentence next to the first sentence area (1154) and re-executing from the semantic value value calculation step (S3546c) if the semantic value value is less than the predetermined threshold value.

[0307] The calculation of the semantic value value can be done cumulatively, because natural language understanding is a method that is not necessarily applied only to the single sentence that has had OCR performed; it can be done in a way that it is reapplied to a larger area of ​​text by combining a sentence that has already had OCR performed previously with the new sentence.

[0308] To this end, step S3548c may iteratively perform OCR on new sentence areas, including subsequent sentences that occur after the delimiter of the first sentence area.

[0309] Of course, if the semantic value value does not reach a predetermined threshold even after OCR is performed up to the last sentence, it can be detected that there is no text consisting of the entire character.

[0310] In this manner, by performing the steps described above in this disclosure, a computing device that processes a video (image) of an object can ascertain various information about the object detected in the video (image).

[0311] For ease of explanation, the components shown in the drawings are exemplified as being implemented in one computing device, for example, a mobile terminal, but it is obvious that the computing device (100) executing the method of the present invention can be configured as a mechanism in which multiple devices cooperate with each other. Therefore, it is obvious that each step of the method of the present invention described above can be executed not only by a mobile terminal but also by a gimbal (300) having a built-in communication unit and processor, but can also be executed directly by one computing device or by the one computing device supporting the execution by another computing device that cooperates with the one computing device.

[0312] As described above, the method and apparatus of the present disclosure, in all its embodiments and modified examples, is capable of recognizing and tracking one or more objects using video (images), actively acquiring information about the objects and the environment, and in particular, is capable of acquiring information about the state of an object from a long distance using video (images), is capable of identifying objects with which an object interacts using video (images), and is capable of acquiring higher resolution detailed video (images) of portions of an object from a long distance, and is capable of grasping from a long distance characters printed on an object or output using other means such as a display, thereby enabling long distance input using video (images) in a portable computing device.

[0313] Based on the above description of various embodiments of the present disclosure, a person skilled in the art can clearly understand that the method and / or process of the present invention and its steps can be realized by hardware, software, or any combination of hardware and software suitable for a particular application. The hardware can include a general-purpose computer and / or a dedicated computing device, or a specific computing device or a particular aspect or component of a specific computing device. The process can be realized by one or more processors, such as microprocessors, controllers, such as microcontrollers, embedded microcontrollers, microcomputers, arithmetic logic units (ALUs), digital signal processors, such as programmable digital signal processors, or other programmable devices, having internal and / or external memory. Additionally or alternatively, the processes may be performed by an application specific integrated circuit (ASIC), a programmable gate array, such as a field programmable gate array (FPGA), a programmable logic unit (PLU), or a programmable array logic (PAL), or any other device capable of executing and responding to instructions, any other device configured to process electronic signals, or a combination of devices. The processing device may execute an operating system (OS) and one or more software applications that run on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software.For ease of understanding, a single processing device is described in some places, but one of ordinary skill in the art will appreciate that a processing device can include multiple processing elements and / or multiple types of processing elements. For example, a processing device can include multiple processors or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0314] Software may include computer programs, codes, instructions, or any combination of one or more of these, capable of configuring or instructing a processing device to perform a desired operation, either individually or collectively. The software and / or data may be embodied, either permanently or temporarily, in a particular type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, to be analyzed by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored and executed in a distributed manner on computer systems linked via a network. The software and data may be stored on one or more machine-readable recording media.

[0315] Furthermore, the subject matter of the technical solution of the present invention or a portion contributing to the prior art may be embodied in the form of program instructions that can be executed by various computer components and recorded on a machine-readable medium. The machine-readable medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions recorded on the machine-readable recording medium may be those specially designed and constructed for the embodiments, or those known and available to those of ordinary skill in the computer software arts. Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, Blu-rays, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include those that can be written using a structured programming language such as C, an object-oriented programming language such as C++, or a higher-level or lower-level programming language (assembly language, hardware technology language, and database programming language and technology), which can be stored and compiled or interpreted to be executed on any one of the aforementioned devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing program instructions, including not only machine code, bytecode, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0316] Thus, in one aspect of the invention, the methods and combinations of methods described above can be implemented as executable code that performs the steps when executed by one or more computing devices. In another aspect, the methods can be implemented as a system that performs the steps, and the methods can be distributed in various ways across multiple devices, or all functions can be integrated into one dedicated, stand-alone device or other hardware. In yet another aspect, the means for performing the steps associated with the processes described above can include any of the hardware and / or software described above. All such sequential combinations and combinations are intended to be within the scope of this disclosure.

[0317] For example, the hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, or vice versa. The hardware device may include a processor, such as an MPU, a CPU, a GPU, or a TPU, coupled to a memory, such as a ROM / RAM, for storing program instructions, and configured to execute the instructions stored in the memory, and may include a communication unit for exchanging signals with an external device. The hardware device may include a keyboard, a mouse, or other external input device for receiving instructions created by a developer.

[0318] The present invention has been described above using specific details such as concrete components, limited examples, and drawings. However, this is merely provided to facilitate a general understanding of the present invention, and the present invention is not limited to the above-described embodiments. A person having ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from such descriptions.

[0319] Therefore, the spirit of the present invention is not limited to the above-described embodiment, but includes not only the claims attached to this disclosure, but also all modifications equivalent to or equivalent to the claims. For example, suitable results can be obtained even if the described techniques are performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted by other components or equivalents. As described above, the equivalent or modified equivalently includes, for example, a logically equivalent method that produces the same result as when the method according to the present invention is carried out, so the true meaning and scope of the present invention should not be limited by the above examples, but should be understood in the broadest sense permissible by law. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0320] As described above, the related content has been described based on the best mode for carrying out the invention. [Industrial Applicability]

[0321] The present invention can be used in devices, systems, etc. that process images acquired from an imaging device.

Claims

1. 1. A method of processing an image, performed by a computing device including a processor, the method comprising: Acquiring an image; Obtaining, from the video (image), analysis information corresponding to objects contained in the video (image) using an object analysis model; and obtaining characters contained in the object from analysis information corresponding to the object using an OCR model; Including, The OCR model is determining whether text is displayed on a surface of the object; extracting at least one image sample from the object on which the character is displayed; determining from said at least one image sample a boundary of an area in which text is displayed; obtaining an image pattern located at least on the boundary line or a boundary point that is a point belonging to the boundary line as a boundary marker from the at least one image sample; A step of generating a character display image (image) that is an image (image) including the boundary marker as a partial image (image) included in an image (image) of an object in which the character is displayed; and performing OCR on the character display image; Execute method.

2. In claim 1, The image sample is an image pattern that exists at at least one of the boundary line or the boundary point; and The video (image) pattern includes at least one of a part of a character, a boundary of the character, a part of the character, or a background. method.

3. In claim 1, The boundary marker is a beginning marker that is a video (image) pattern that corresponds to the first character of the characters; or an end marker, which is a video (image) pattern corresponding to the last character of said characters; At least one of the following is included: method.

4. In claim 3, The step of obtaining an image pattern located at at least one of the boundary line or the boundary point from the at least one image sample as the boundary marker includes: Using a tracking controller, acquiring an image of a character including the leading marker and having a character recognition rate equal to or higher than a threshold; and determining whether the image of the character includes the end marker; Including, method.

5. In claim 4, determining the image (image) of the character as the image (image) of the character display when the image (image) of the character includes the end marker; Further comprising: method.

6. In claim 4, if the character image does not include the end marker, acquiring an additional character image using the tracking controller, the additional character image including a marker next to the last marker among the boundary markers included in the character image and the character recognition rate being equal to or greater than the threshold; combining the additional character image with the character image to generate a combined character image; and determining the image (image) of the combined characters as the character display image (image) if the end marker is included in the image (image) of the combined characters; Further comprising: method.

7. In claim 1, After the step of determining whether or not a character is displayed on the surface of the object, using a tracking controller to capture an image of the object that includes the beginning of the character and has a character recognition rate equal to or greater than a threshold; performing said OCR on an area of ​​a first sentence in an image of the object; and performing natural language understanding (NLU) on the first text, which is a primary result of the OCR, to calculate a first semantic value value; Including, method.

8. In claim 7, determining the first text as a result text that is a result of the OCR if the first semantic value value is equal to or greater than the threshold value; Further comprising: method.

9. In claim 7, performing the OCR on an area of ​​text subsequent to the first area of ​​text if the first semantic value value is less than the threshold; performing natural language understanding on the second text, which is a primary result of the OCR, to calculate a second semantic value value, which is a value of semantic value; and determining the second text as a result text that is a result of the OCR if the second semantic value value is equal to or greater than the threshold value; Further comprising: method.

10. In claim 1, The computing device is in communication with the imaging device and the gimbal; The step of acquiring the image (image) includes: using a tracking controller to actuate one or more rotational axes of the gimbal to control the orientation of the imager; and acquiring the image by zooming in or out of the image capture device using the tracking controller; Including, method.

11. A non-transitory computer readable medium containing a computer program, the computer program causing a computing device to perform a method for processing an image, the method comprising: Acquiring an image; Obtaining, from the video (image), analysis information corresponding to objects contained in the video (image) using an object analysis model; and obtaining characters contained in the object from analysis information corresponding to the object using an OCR model; Including, The OCR model is determining whether text is displayed on a surface of the object; extracting at least one image sample from the object on which the character is displayed; determining from said at least one image sample a boundary of an area in which text is displayed; obtaining an image pattern located at least on the boundary line or a boundary point that is a point belonging to the boundary line as a boundary marker from the at least one image sample; A step of generating a character display image (image) that is an image (image) including the boundary marker as a partial image (image) included in an image (image) of an object in which the character is displayed; and performing OCR on the character display image; Execute A non-transitory computer readable medium containing a computer program.

12. In a computing device, A processor; and Communications Department; Including, The processor, Acquire video (images), Using an object analysis model, obtain from the video (image) analysis information corresponding to objects contained in the video (image); and Using an OCR model, obtain characters contained in the object from analysis information corresponding to the object; The OCR model is determining whether or not a character is displayed on a surface of the object; Extracting at least one image sample from the object on which the character is displayed; determining from said at least one video (image) sample a boundary of an area in which text is displayed; Obtaining an image pattern located at least on one of the boundary lines or boundary points that are points belonging to the boundary line as a boundary marker from the at least one image sample; Generate a character display image (image) that is an image (image) including the boundary marker as a partial image (image) included in an image (image) of an object in which the character is displayed; and Executing OCR on the character display image (image); Computing device.

Images