Method of processing images acquired from photographing device in linkage with computing device and system using same

Abstract

An image processing method according to various embodiments of the present invention is performed by a computing device including a processor, the method comprising the steps of: acquiring an image; obtaining analysis information corresponding to an object included in the image from the image by using an object analysis model; and acquiring characters included in the object from analysis information corresponding to the object by using an optical character recognition model. The abstract drawing can be figure 2.

Description

Technical Field

[0001] The present invention discloses a method for processing an image acquired from a camera and a system using the same. Specifically, according to the method of the present invention, a computing device acquires an overall image from a camera that is integrated with the computing device or linked to the computing device, detects one or more objects displayed in the overall image, and performs classification of the detected objects by calculating the categories thereof, thereby generating detailed classification information including the characteristics and states of the objects as analysis results of the respective objects. Background Art

[0002] Generally, a portable computing device is a device equipped with a processor, a display, a microphone and a speaker, a part of which can be used as a communication device, i.e., a portable terminal. A portable terminal receives user commands through input devices such as a keyboard and a touch display that the user needs to touch. However, with the development of speech recognition technology and computer vision technology, user commands can be received remotely through input devices such as microphones and cameras and interaction can be performed according to the commands.

[0003] However, since portable computing devices themselves do not have mobility, the range of input and output is limited to their physical location.

[0004] As an example of the physical limitations of the input device mounted on the portable computing device, the camera of the contactless input device is mounted on the front or back of the mobile phone and has a limited field of view. In order for the camera to capture the desired image, the user needs to hold the camera with his hand to directly change the composition (field of view; FOV). As another example of such physical limitations, the microphone mounted on the portable computing device reduces the receiving sensitivity depending on the direction of the sound source.

[0005] In addition to input, output is also limited by physical location. As one example, a touch display as an output device usually has a shape close to a plane and is attached to a part of the six faces of a portable terminal. The user needs to hold the mobile phone and use it so that the touch display of the portable terminal faces the direction of his face. As another example, for an infrared projector used to obtain a three-dimensional shape, since the irradiation direction and angle of the infrared ray are limited to a direction range of the portable terminal, the user also needs to hold the portable terminal and adjust its irradiation direction according to the guidance. In addition, the technical documents related to this include Korean Patent Publication No. 10-2011-0032244, No. 10-2019-0085464, No. 10-2019-0074011, No. 10-2019-0098091, No. 10-2019-0106943, and No. 10-2018-0109499. Moreover, the related non-technical literatures include Y.Zhou et al., "Learning to Reconstruct 3DManhattanWireframes From a Single Image," 2019IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp.7697-7706, doi:10.1109 / ICCV.2019.00779. (Non-technical literature 1), Shin, D., & Kim, I. (2018). Deep Image Understanding Using Multilayered Contexts. Mathematical Problems in Engineering, 2018, 1-11. https: / / doi.org / 10.1155 / 2018 / 5847460 (non-technical literature 2), Mo, K., Zhu, S., Chang, AX, Yi, L., Tripathi, S., Guibas, LJ, & Su, H. (2019). PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding.In2019IEEE / CVF Conference on ComputerVision and Pattern Recognition(CVPR).IEEE.https: / / doi.org / 10.1109 / cvpr.2019.00100 (non-technical literature 3), Babaee, M., Li, L., & Rigoll, G. (2019). Person identification from partial gait cycle using fully convolutional neural networks. Neurocomputing, 338, 116-125. (non-technical literature 4), Muhammad, UR, Svanera, M., Leonardi, R., & Benini, S. (2018). Hair detection, segmentation, and hairstyle classification in the wild. Image and Vision Computing, 71, 25-37. https: / / doi.org / 10.1016 / j.imavis.2018.02.001 (non-technical literature 5), Mougeot, G., Li, D., & Jia, S. (2019). A Deep Learning Approach for Dog Face Verification and Recognition. In PRICAI 2019: Trends in ArtificialIntelligence(pp.418-430).Springer International Publishing.https: / / doi.org / 10.1007 / 978-3-030-29894-4_34 (non-technical literature 6), Raduly, Z., Sulyok, C., Vadaszi, Z., & Zolde, A. (2018). Dog Breed IdentificationUsing Deep Learning.In 2018IEEE 16thInternational Symposium on Intelligent Systems and Informatics(SISY).IEEE.https: / / doi.org / 10.1109 / sisy.2018.8524715(Non-technical document 7), Wu, Z., Yao, T., Fu, Y., & Jiang, Y.-G. (2017). Deep learning for video classification and captioning.InFrontiers of Multimedia Research(pp.3-29).ACM.https: / / doi.org / 10.1145 / 3122865.3122867 (Non-technical literature 8), Wu, C.-Y., Girshick, R., He, K., Feichtenhofer, C., & Krahenbuhl, P. (2020). A Multigrid Method for Efficiently Training Video Models. In 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. https: / / doi.org / 10.1109 / cvpr42600.2020.00023 (Non-technical literature 9), Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, SW (2018). Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features. IEEE Access, 6, 1155-1166. https: / / doi.org / 10.1109 / access.2017.2778011 (Non-technical literature 10). . Summary of the invention

[0006] Technical issues

[0007] The present invention provides a solution that can overcome the limitations related to image input among the limitations of the portable computing devices of the prior art described above, that is, a pan-tilt head is combined with an image capture device such as a camera linked to the portable computing device and the computing device controls the pan-tilt head, thereby allowing the image capture device to be rotated using more than one axis to more dynamically receive and process images.

[0008] The object of the present invention is to provide an image processing method that can be remotely input through images. That is, the present invention can solve the problems of the prior art to enable a portable computing device to recognize and track an object in an image taken by a camera, etc., and to dynamically obtain information about the object and the environment. In particular, the relative position information and spatial information of the object can be grasped based on the image in a coordinate system centered on the system of the present invention to confirm the position of the object in space.

[0009] Technical Solution

[0010] In order to achieve the above-mentioned objectives, the image processing method of the present invention is executed by a computing device including a processor, and the method includes the following steps: acquiring an image; acquiring analysis information corresponding to an object included in the image from the image using an object analysis model; and acquiring characters included in the object from the analysis information corresponding to the object using an optical character recognition (OCR) model.

[0011] Alternatively, the optical character recognition model may be used to perform the following steps: determining whether characters are displayed on the surface of the object; and performing optical character recognition on the characters if it is determined that the characters are displayed on the surface of the object.

[0012] Alternatively, the step of performing optical character recognition on the character may include the following steps: extracting at least one image sample from the object displaying the character; determining a boundary line of a character display area from the at least one image sample; generating a character display image based on at least one of the boundary points located on or within the boundary line in the at least one image sample; and performing optical character recognition on the character display image.

[0013] Alternatively, the image sample is an image pattern existing at at least one position of the boundary line or the boundary point, and the image pattern may include at least one of a partial character, a boundary portion of the character, a part of the character, or a background.

[0014] Alternatively, the step of generating a character display image based on at least one of the boundary points located on the boundary line or within the boundary line in the at least one image sample may include the following steps: acquiring an image pattern located on at least one of the boundary line or the boundary points in the at least one image sample as a boundary marker; and generating a character display image as a partial image included in the object image displaying the character, the character display image being an image including the boundary marker.

[0015] Alternatively, the boundary marker may include at least one of a start marker or an end marker, the start marker being an image pattern corresponding to a start character of the character, and the end marker being an image pattern corresponding to an end character of the character.

[0016] Alternatively, the step of acquiring an image pattern located at least one of the boundary line or the boundary point in the at least one image sample as the boundary marker may include the following steps: using a tracking controller to acquire a character image including the start mark and having a character recognition rate above a threshold; and determining whether the character image includes the end mark.

[0017] Alternatively, the method may further include the following step: if the character image includes the end mark, determining the character image as the character display image.

[0018] Alternatively, the following steps may also be included: if the character image does not include the end mark, using the tracking controller to obtain an additional character image including the next mark of the last mark from the boundary marks included in the character image and the character recognition rate is above the threshold; merging the additional character image into the character image to generate a merged character image; and if the merged character image includes the end mark, determining the merged character image as the character display image.

[0019] Alternatively, the step of performing the optical character recognition on the character may include the following steps: using a tracking controller to obtain the object image including the starting point of the character and the character recognition rate being above a threshold; performing the optical character recognition on an initial text area of ​​the object image; and performing natural language understanding (NLU) on a first text as a first result of the optical character recognition to calculate a first semantic value as a semantic numerical value.

[0020] Alternatively, the method may further include the following step: if the first semantic value is greater than the threshold value, determining the first text as a result text obtained as a result of the optical character recognition.

[0021] Alternatively, the following steps may also be included: if the first semantic value is less than the threshold, performing the optical character recognition on the next text area of ​​the initial text area; performing natural language understanding on a second text which is a first result of the optical character recognition to calculate a second semantic value which is a semantic numerical value; and if the second semantic value is above the threshold, determining the second text as a result text which is a result of the optical character recognition.

[0022] Alternatively, the computing device is linked with the shooting device and the gimbal, and the step of acquiring the image may include the following steps: using a tracking controller to operate one or more rotation axes of the gimbal to control the direction of the shooting device; and using the tracking controller to acquire the image by zooming in (zoom-in) or out (zoom-out) of the shooting device.

[0023] In order to achieve the above-mentioned purpose, the non-transitory computer-readable medium of the present invention includes a computer program, which enables a computing device to execute a method for processing an image, and the method may include the following steps: acquiring an image; acquiring analysis information corresponding to an object included in the image from the image using an object analysis model; and acquiring characters included in the object from the analysis information corresponding to the object using an optical character recognition model.

[0024] In order to achieve the above-mentioned purpose, the computing device of the present invention includes: a processor; and a communication unit, and the processor can execute the following steps: acquiring an image; acquiring analysis information corresponding to the object included in the image from the image using an object analysis model; and acquiring the characters included in the object from the analysis information corresponding to the object using an optical character recognition model.

[0025] Effects of the Invention

[0026] According to an exemplary embodiment of the present invention, images can be used to identify and track one or more objects, and information about the objects and the environment can be actively obtained. In particular, state information of the objects can be remotely obtained through images, and objects interacting with the objects can be determined based on the images. Higher-resolution detailed images of parts of the objects can be remotely obtained, and characters printed on the objects or output by other devices such as displays can be remotely grasped. Therefore, it has the effect of enabling portable computing devices to use images for remote input. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings used to illustrate the embodiments of the present invention are only a part of the embodiments of the present invention. Therefore, an ordinary technician in the technical field to which the present invention belongs (hereinafter referred to as "ordinary technicians") can obtain other drawings based on the following drawings and easily derive the present invention.

[0028] Figure 1 The present invention is a conceptual diagram briefly showing an exemplary structure of a computing device for executing an image processing method (hereinafter referred to as “image processing method”) according to an embodiment of the present invention.

[0029] Figure 2 The system for executing the image processing method according to one embodiment of the present invention is an illustrative conceptual diagram of the overall hardware and software architecture including a computing device, a shooting device and a gimbal.

[0030] Figure 3 To illustrate an exemplary flow chart of an image processing method according to an embodiment of the present invention, Figure 4 The figure is an exemplary block diagram showing modules for executing various steps of an image processing method according to an embodiment of the present invention.

[0031] Figure 5The figure is an exemplary block diagram showing a machine learning model required for executing a module of an image processing method according to an embodiment of the present invention.

[0032] Figures 6a to 6d FIG. 1 is an exemplary flow chart showing how the image processing method of the present invention can be used to detect the ground plane of an object.

[0033] Figure 7a An exemplary diagram showing object segmentation obtained by an image processing method according to an embodiment of the present invention.

[0034] Figure 7b The figure is an example diagram for explaining the steps of detecting more than two ground planes by an image processing method according to an embodiment of the present invention.

[0035] Figure 7c The figure is a conceptual diagram for explaining the method of generating and using a reference plane circle and a measurement plane circle in an image processing method according to an embodiment of the present invention.

[0036] Figures 8a to 8c The flowchart is an exemplary flowchart showing the manner in which the image processing method of the present invention can be used to determine the target position.

[0037] Figure 9a and Figure 9b The flowchart is an exemplary flowchart showing a method for controlling the orientation of a camera in the image processing method of the present invention.

[0038] Figures 10a to 10c FIG. 1 is an exemplary flow chart showing the manner in which the image processing method of the present invention can be used to perform optical character recognition.

[0039] Figures 11a to 11d It is an illustrative diagram for explaining the execution mode of optical character recognition in the image processing method of the present invention. DETAILED DESCRIPTION

[0040] All prior art documents cited in this specification are as shown in the disclosure of this specification, and their entirety is used as a reference. Unless otherwise defined, all terms used herein, including technical terms or scientific terms, have the same meanings as those commonly understood by ordinary technicians in the technical field to which the present invention belongs. Terms defined in commonly used dictionaries should be interpreted as having the same meaning as the context of the relevant technology, and should not be interpreted as idealized or overly formalized meanings unless clearly defined in this specification.

[0041] The following detailed description of the present invention is only used to clearly explain the purpose, technical solutions and advantages of the present invention, and reference should be made to the accompanying drawings that show specific embodiments that can implement the present invention as examples. Such embodiments are used for detailed description so that ordinary technicians in the technical field to which the present invention belongs can fully understand and easily implement the present invention. In the process of describing with reference to the accompanying drawings, the same structural elements are given the same figure numbers regardless of the figure numbers, and their repeated descriptions will be omitted.

[0042] The characteristic structure or function description of the embodiment is for illustrative purposes only and can be implemented through various implementations. Therefore, the embodiment is not limited to a specific implementation, and the scope of this specification includes the modified technical solutions, equivalent technical solutions or alternative technical solutions belonging to the technical ideas.

[0043] The terms "first" or "second" may be used to describe multiple structural elements, but such terms are only used to distinguish one structural element from other structural elements and do not limit any order. For example, the first structural element may be named the second structural element, and similarly, the second structural element may also be named the first structural element.

[0044] When a structural element is expressed as being “connected to” another structural element, although it may be directly connected or coupled to the other structural element, it may also be understood that there are other structural elements in between.

[0045] Unless the context clearly indicates otherwise, expressions in the singular include expressions in the plural. In this specification, terms such as "including" or "having" are used only to refer to the existence of the recorded features, numbers, steps, operations, structural elements, components or their combinations, and do not preclude the existence or additional possibility of one or more other features, numbers, steps, operations, structural elements, components or their combinations.

[0046] Furthermore, "part" or "a portion" of an object refers to a portion of the object, not the entire object, but should be understood to include the entire object unless the context clearly indicates a different meaning. This is similar to the concept that a part of a set includes the set itself.

[0047] In this specification, "module" refers to hardware that can perform the functions and operations of each name described in this specification, and can also refer to computer program codes that can perform specific functions and operations, or can also refer to a recording medium loaded with computer program codes that can perform specific functions and operations. In other words, a module refers to a functional and / or structural combination of hardware used to implement the technical ideas of the present invention and / or software used to drive the hardware.

[0048] Specifically, a "model" refers to a function that calculates output data based on input data trained by machine learning. This "model" is a data structure or function that can be used in the above-mentioned "module".

[0049] However, since some ordinary technicians in the field of artificial intelligence applications have the habit of using "module" and "model" interchangeably, "module" and "model" may also be used interchangeably in this specification so that ordinary technicians in the technical field to which the present invention belongs can easily understand and not confuse the concepts with each other.

[0050] In this specification, "training" and "learning" refer to terms for performing machine learning through step-based calculations. It should be understood by ordinary technicians in the technical field to which the present invention belongs that they are not used to refer to mental effects such as human educational activities. In the same meaning as commonly used in the field of statistics, "machine learning" is generally used to refer to a series of creation processes of a target function (f) (mapping) of an input variable (X) to an output variable (Y). "Predict" is used to refer to the process of calculating an output variable based on an input variable through a target function, and "mapping" refers to reasonably reducing the difference between the actual value and the predicted value. However, the reason for reasonably reducing the difference rather than minimizing the difference is that there is an overfitting problem in optimization, that is, when actual data that deviates from the training data is applied, there is a hidden danger that may lead to inaccurate predictions. Therefore, in order to solve this problem, it is necessary to seek appropriate empirical means.

[0051] Also, the term "inference" refers to the process of calculating output data based on input data through a machine learning model, and in particular, is used to refer to mechanically imitating human mental functions. Similar to inference, in this specification, machine "analysis" is used to refer to mechanically imitating human mental functions.

[0052] In this specification, "Manhattan space" is the same as the content disclosed in the paper Y.Zhou et al.,"Learning toReconstruct3D Manhattan Wireframes From a Single Image,"2019IEEE / CVFInternationalConference on Computer Vision(ICCV),Seoul,Korea(South),2019,pp.7697-7706,doi:10.1109 / ICCV.2019.00779.

[0053] Furthermore, the present invention includes all combinations of the embodiments described in this specification as much as possible. Although the multiple embodiments of the present invention are different from each other, they should not be understood to be mutually exclusive. For example, the specific shapes, structures and characteristics described herein are related to an embodiment and can be implemented by different embodiments without departing from the scope of the present invention. Moreover, in each disclosed embodiment, the position or configuration of each structural element can be changed in many ways without departing from the scope of the present invention, and therefore, the following detailed description should not be understood as limiting the meaning. In the accompanying drawings, similar reference numerals may also refer to the same or similar functions through multiple levels.

[0054] In this specification, unless otherwise specified or clearly not contradictory in context, items expressed in the singular may include the plural unless otherwise required in context. In addition, in the process of describing the present invention, when it is judged that the detailed description of the known structure or function may unnecessarily confuse the main idea of ​​the present invention, its detailed description will be omitted.

[0055] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention.

[0056] Figure 1 The diagram briefly illustrates an exemplary structure of a computing device for executing an image processing method according to an embodiment of the present invention.

[0057] Reference Figure 1 The computing device 100 of one embodiment of the present invention includes a communication unit 110 and a processor 120 , and can communicate with an external computing device (not shown) directly or indirectly through the communication unit 110 .

[0058] Specifically, the computing device 100 can utilize a combination of existing computer hardware (e.g., a computer; a device that may include a processor, memory, storage, input device and output device, other structural elements of existing computing devices, etc.; electronic communication devices such as routers and switches; electronic information storage systems such as network-attached storage (NAS) and storage area network (SAN)) and computer software (i.e., instructions that enable the computing device to perform functions in a specific manner) to achieve the desired system functions. In addition to recording devices such as hard disks and universal serial bus (USB) memory, the memory may include network-connected storage devices such as cloud servers. Among them, the memory may be DDR2, DDR3, DDR4, SDP, DDP, QDP, magnetic hard disk, flash memory, etc., but is not limited thereto.

[0059] The communication unit 110 of the computing device can send and receive requests and responses with other linked computing devices, for example, it can send and receive requests and responses with a portable terminal, etc. As an example, the request and response can be implemented through the same Transmission Control Protocol (TCP) session, but is not limited to this. For example, it can also be sent and received through a User Datagram Protocol (UDP) datagram.

[0060] Specifically, the communication unit 110 may be a communication module including a communication interface. For example, the communication interface may include wireless LAN (WLAN), wireless hotspot (WiFi), direct connection (Direct), Digital Living Network Alliance (DLNA), wireless broadband (WiBro), World interoperability for microwave access (WiMax), High Speed ​​Downlink Packet Access (HSDPA), 4G, 5G and other wireless Internet interfaces and Bluetooth (Bluetooth TM), Radio Frequency IDentification (RFID), Infrared Data Association (IrDA), Ultra-Wide Band (UWB), ZigBee, Near Field Communication (NFC), and other short-range communication interfaces. In addition, the communication interface can represent all interfaces that can communicate with the outside (for example, wired interfaces).

[0061] For example, the communication unit 110 can send and receive data with other computing devices through such an appropriate communication interface. Moreover, in a broad sense, the communication unit 110 includes a keyboard, a mouse, a touch sensor, an input unit of a touch display screen, a microphone, a camera or a laser radar (LIDAR), a radar, a switch, a button, an external input device such as a joystick, a sound card, a graphics card, a printing device, and a display for transmitting instructions or instructions, for example, and may include an external output device such as a display unit of a numerical control display screen or be linked with it. For example, in order for the computing device to display an appropriate user interface to the user of the portable terminal to interact with the user, the computing device 100 may be built into a display device or linked with an external display device through the communication unit 110. For example, such a display device may be a touch screen capable of touch input. Objects such as fingers, styluses, etc. that contact or approach the display can be detected based on the capacity, guidance, or optics of the touch screen, and their detected positions on the display can be determined.

[0062] The input device may also include a microphone. The types of microphones may include dynamic microphones, condenser microphones, etc., and microphones with characteristics such as omnidirectional, unidirectional, and super-directional may be used. Although beamforming microphones and microphone arrays may also be used, they are not limited thereto. A microphone array refers to two or more microphones used to detect the direction of a sound source.

[0063] The output device may also include a speaker. The types of speakers may include a non-directional speaker, a directional speaker, a super-directional speaker using ultrasonic waves, etc., but are not limited thereto.

[0064] Furthermore, the processor 120 of the computing device may include a microprocessor (MPU, micro processing unit), a central processing unit (CPU, central processing unit), a graphics processing unit (GPU, graphics processing unit), a neural network processor (NPU, neural processing unit), an application-specific integrated circuit (ASIC), a complex instruction set (CISC), a reduced instruction set computer (RISC), a field programmable gate array (FPGA), a system on chip (SOC) or a tensor processing unit (TPU, tensor processing unit), a cache memory (cache memory), a data bus and other hardware structures. Furthermore, it may also include a software structure for executing an operating system and a specific purpose application. According to an embodiment of the present invention, the processor 120 may perform operations for learning neural networks of various models. The processor 120 may perform input data processing based on deep learning (DL, deep learning), feature extraction based on input data, error calculation, updating weight values ​​of the neural network using back propagation and other calculations for learning. At least one of the central processing unit (CPU), general purpose graphics processing unit (GPGPU, general purpose graphics processing unit) and / or tensor processing unit (TPU) of the processor 110 may process the learning of network functions. For example, a central processing unit (CPU) and a general purpose graphics processing unit (GPGPU) may process together learning using a network function and data classification using a network function. Furthermore, in one embodiment of the present invention, processors of multiple computing devices may be used together to process learning using a network function and data classification using a network function. Furthermore, in one embodiment of the present invention, a computer program executed by the computing device 100 refers to a program executable by a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0065] Throughout the text of this specification, computational model, neural network, network function, neural network (neural network) can be used as the same meaning. A neural network can be composed of a collection of computing units connected to each other, which can be generally referred to as nodes. Such a node can also refer to a neuron. A neural network includes more than one node. The nodes (or neurons) constituting a neural network can be connected to each other through more than one link.

[0066] In a neural network, more than one node connected by a link can form an input node and an output node relationship with each other. The concepts of input node and output node are relative, and any node that forms an output node relationship with one node can form an input node relationship with another node, and vice versa. As described above, the relationship between input nodes and output nodes can be generated around links. More than one output node can be connected to an input node through a link, and vice versa.

[0067] In the relationship between an input node and an output node connected by a link, the data of the output node may determine its value based on the input data of the input node. The link connecting the input node to the output node may have a weight. The weight is variable and may be changed by a user or an algorithm in order to perform the desired function of the neural network. For example, when more than one input node is connected to an output node through each link, the output node may determine the output node value based on the input value of the input node connected to the output node and the weight set in the link corresponding to each input node.

[0068] As described above, in a neural network, one or more nodes may be connected by one or more links and form a relationship between an input node and an output node in the neural network. In a neural network, the characteristics of the neural network can be determined by the number of nodes and links, the correlation between the nodes and the links, and the weighted values ​​assigned to the links. For example, when there are two neural networks with the same number of nodes and links but different weighted values ​​of the links, it can be recognized that the two neural networks are different from each other.

[0069] A neural network may be composed of a collection of more than one nodes. A partial collection of nodes constituting a neural network may constitute a layer. Among the nodes constituting a neural network, a portion may constitute a layer based on the distance from an initial input node. For example, a collection of nodes whose distance from an initial input node is n may constitute n layers. The distance from the initial input node may be defined based on the minimum number of links that must be passed to reach the corresponding node from the initial input node. However, the definition of the layer is arbitrary for the purpose of illustration, and the number of layers in a neural network may be defined by a method different from that described. For example, the layer of a node may also be defined based on the distance from the final output node.

[0070] Among the nodes in the neural network, the initial input node refers to one or more nodes that directly input data without links in the relationship with other nodes. Alternatively, it refers to a node that does not have other input nodes connected by links in the relationship with a node based on links in the neural network. Similarly, among the nodes in the neural network, the final output node refers to one or more nodes that do not have output nodes in the relationship with other nodes. And, hidden nodes refer to nodes that constitute the neural network other than the initial input node and the final output node.

[0071] In a neural network of one embodiment of the present invention, the number of nodes in the input layer may be the same as the number of nodes in the output layer, which refers to a neural network in which the number of nodes gradually decreases as the input layer progresses to the hidden layer and then increases again. Furthermore, in a neural network of yet another embodiment of the present invention, the number of nodes in the input layer may be less than the number of nodes in the output layer, which refers to a neural network in which the number of nodes gradually decreases as the input layer progresses to the hidden layer. Furthermore, in a neural network of another embodiment of the present invention, the number of nodes in the input layer may be greater than the number of nodes in the output layer, which refers to a neural network in which the number of nodes gradually increases as the input layer progresses to the hidden layer. In another embodiment of the present invention, the neural network may be a combination of the aforementioned neural networks.

[0072] A deep neural network (DNN) refers to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. If a deep neural network is used, the latent structure of the data can be grasped. That is, the latent structure of photos, text, video, speech, and music can be grasped (for example, what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the speech, etc.). A deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), an auto encoder, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a twin neural network, a generative adversarial network (GAN), etc. The description of the deep neural network is only an example, and the present invention is not limited to this.

[0073] A neural network can learn by at least one of supervised learning, unsupervised learning, semi supervised learning, or reinforcement learning. Learning of a neural network refers to the process of applying knowledge that a neural network uses to perform a specific task to the neural network.

[0074] Neural networks can learn in the direction of minimizing output errors. Learning of neural networks refers to the following process, that is, repeatedly inputting learning data into the neural network to calculate the output error and calibration error of the neural network relative to the learning data, and backpropagating the error of the neural network from the output layer along the output layer direction in the direction of reducing the error to update the weight value of each node of the neural network. In the case of supervised learning, learning data marked with the correct answer (i.e., labeled learning data) is used for each learning data, and in the case of unsupervised learning, each learning data may not be marked with the correct answer data. That is, for example, in supervised learning related to data classification, the learning data refers to data with a category marked in each learning data. The labeled learning data is input into the neural network, and the output (category) of the neural network can be compared with the label of the learning data to calculate the error. As another example, in unsupervised learning related to data classification, the learning data as input can be compared with the output of the neural network to calculate the error. In the neural network, the calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer), and the connection weight value of each node of each layer of the neural network can be updated by backpropagation. The updated connection weights of each node may be changed based on the learning rate. The calculation of the neural network with respect to the input data and the back propagation of the error may constitute a learning cycle. Different learning rates may be applied according to the number of repetitions of the learning cycle of the neural network. For example, in the early stages of learning of the neural network, a high learning rate may be used to allow the neural network to quickly ensure a specified level of performance to improve efficiency, and in the later stages of learning, a low learning rate may be used to improve accuracy.

[0075] In the learning of neural networks, the learning data can usually be a partial set of actual data (i.e., data processed using the learned neural network), so there may be a learning cycle in which the errors of the learning data decrease while the errors of the actual data increase. Overfitting refers to the phenomenon that the errors of the actual data increase due to over-learning of the learning data. For example, as an overfitting phenomenon, if a neural network that learns cats by observing yellow cats cannot recognize cats when observing cats other than yellow. Overfitting can lead to an increase in errors in machine learning algorithms. In order to prevent such overfitting, a variety of optimization methods can be used. In order to prevent overfitting, methods such as increasing learning data, regularization, closing the discarding of some nodes of the network during the learning process, and applying a batch normalization layer can be applied.

[0076] An embodiment of the present invention discloses a computer-readable medium storing a data structure.

[0077] Data structure refers to data organization, data management, and data storage that can effectively access and modify data. Data structure refers to data organization used to solve specific problems (for example, data retrieval, data storage, and data modification in the shortest time). Data structure is designed to support specific data processing functions and can also be defined as physical or logical relationships between data units. Logical relationships between data units may include user-defined connection relationships between data units. Physical relationships between data units may include actual relationships between data units physically stored in computer-readable storage media (for example, permanent storage devices). Specifically, data structures may include data sets, relationships between data, and functions or instructions that can be applied to data. Effectively designed data structures can minimize the use of computing device resources to perform operations. Specifically, computing devices can improve the efficiency of operations, reading, insertion, deletion, comparison, interaction, and retrieval through effectively designed data structures.

[0078] According to the form of the data structure, the data structure can be divided into linear data structure and nonlinear data structure. A linear data structure refers to a structure in which only one data is connected to another data. Linear data structures may include lists, stacks, queues, and deques. A list refers to a series of data sets with an internal order. A list may include a linked list. A linked list refers to a data structure in which data are connected in a row by pointers. In a linked list, a pointer may include connection information for the next data or the previous data. A linked list may be a single linked list, a double linked list, or a circular linked list according to its form. A stack refers to a data arrangement structure that can restrict access to data. A stack may be a linear data structure that can only process (for example, insert or delete) data at one end of the data structure. A stack may be a data structure that stores data in a last-in-first-out manner (LIFO-Last in First Out). A queue refers to a data arrangement structure that can restrict access to data, and unlike a stack, it may be a data structure that stores data in a first-in-first-out manner (FIFO-First in First Out). A deque refers to a data arrangement structure that can restrict access to data, and unlike a stack, it may be a data structure that stores data in a first-in-first-out manner (FIFO-First in First Out). A deque refers to a data structure that can process data at both ends of a data structure.

[0079] A nonlinear data structure refers to a structure in which multiple data are connected to one data. A nonlinear data structure may include a graph data structure. A graph data structure may be defined by vertices and edges, and an edge may include a line connecting two different vertices. A graph data structure may include a tree data structure. A tree data structure refers to a data structure in which a path connecting two different vertices among multiple vertices included in the tree is one. That is, a graph data structure may be a data structure that does not form a loop.

[0080] In the entire content of this specification, the operation model, neural network, network function, neural network (neural network) can be used as the same meaning. The following unified description is a neural network. The data structure may include a neural network. Moreover, the data structure including the neural network may be stored in a computer-readable medium. The data structure including the neural network may also include pre-processed data used by the neural network for processing, input data of the neural network, weighted values ​​of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions related to each node or layer of the neural network, loss functions used by the neural network for learning, etc. The data structure including the neural network may include any structural elements in the disclosed structure. That is, the data structure including the neural network may include all or any combination of pre-processed data used by the neural network for processing, input data of the neural network, weighted values ​​of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions related to each node or layer of the neural network, loss functions used by the neural network for learning, etc. In addition to the structure, the data structure including the neural network may include any other information for determining the characteristics of the neural network. In addition, the data structure may include all types of data used or generated in the operation process of the neural network, but is not limited to the content. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a collection of interconnected computational units that can be generally referred to as nodes. Such nodes may also be referred to as neurons. A neural network includes more than one node.

[0081] The data structure may include input data of the neural network. The data structure including the input data of the neural network may be stored in a computer-readable medium. The input data of the neural network may include learning data input during the learning process of the neural network and / or input data input to the neural network that has completed learning. The input data of the neural network may include data that has been pre-processed and / or data that has become a pre-processing object. Pre-processing may include a data processing process for inputting data to the neural network. Therefore, the data structure may include data that has become a pre-processing object and data generated based on pre-processing. The data structure is only an example, and the present invention is not limited to this.

[0082] The data structure may include weights of the neural network (in this specification, weights and parameters may be used as the same meaning). Moreover, the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include multiple weights. The weights are variable and may be changed by a user or an algorithm in order to perform the desired function of the neural network. For example, when more than one input node is connected to an output node through each link, the output node may determine the data value output by the output node based on the input values ​​of the input nodes connected to the output node and the weights set in the links corresponding to each input node. The data structure is only an example, and the present invention is not limited to this.

[0083] As a non-limiting example, the weighted values ​​may include weighted values ​​that are variable during the learning process of the neural network and / or weighted values ​​that the neural network has completed learning. The weighted values ​​that are variable during the learning process of the neural network may include weighted values ​​at the beginning of the learning cycle and / or weighted values ​​that are variable during the learning cycle. The weighted values ​​that the neural network has completed learning may include weighted values ​​that have completed the learning cycle. Therefore, the data structure including the weighted values ​​of the neural network may be a data structure including weighted values ​​that are variable during the learning process of the neural network and / or weighted values ​​that the neural network has completed learning. Therefore, the data structure including the weighted values ​​of the neural network includes the weighted values ​​and / or a combination of the weighted values. The data structure is only an example, and the present invention is not limited to this.

[0084] After the serialization process, the data structure including the weighted values ​​of the neural network can be stored in a computer-readable storage medium (e.g., memory, hard disk). Serialization refers to a conversion process that can make the data structure the same or stored in other computing devices and reconstructed for use. As the data structure is serialized, the computing device can send and receive data through the network. The data structure including the weighted values ​​of the serialized neural network can be reconstructed on the same computing device or a different computing device through deserialization. The data structure including the weighted values ​​of the neural network is not limited to serialization. Furthermore, the data structure including the weighted values ​​of the neural network may include a data structure for minimizing the use of computing device resources and improving computing efficiency (e.g., a binary tree (B-Tree), a dictionary tree (Trie), a fork search tree (m-way search tree), a highly balanced binary tree (AVLtree), and a red-black tree (Red-Black Tree) in a nonlinear data structure). The content described is only an example, and the present invention is not limited to this.

[0085] The data structure may include a hyper-parameter of the neural network. Moreover, the data structure including the hyper-parameter of the neural network may be stored in a computer-readable medium. The hyper-parameter may be a variable that can be changed by a user. For example, the hyper-parameter may include a learning rate, a cost function, the number of learning cycle repetitions, a weight initialization (e.g., a range setting of weights that become weight initialization objects), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The data structure is only an example, and the present invention is not limited to this.

[0086] Figure 2 The system for executing the image processing method according to one embodiment of the present invention is an illustrative conceptual diagram of the overall hardware and software architecture including a computing device, a shooting device and a gimbal.

[0087] Reference Figure 2 According to the structure of the method and device of the present invention, the computing device 100 may also include a camera 200, which may be linked to an external camera 200 in a wireless or wired manner, and the computing device 100 may be linked to or include a gimbal 300 in a wireless or wired manner to perform a function for controlling the posture of the camera 200. In order to control the posture of the camera 200, the gimbal 300 may include the camera 200 or be provided with a prescribed mechanism (e.g., an adsorption plate, etc.) for fixing the camera 200.

[0088] In order to control its posture, the gimbal 300 may have more than one rotation axis, and reference may be made to the example in Korean Patent Publication No. 10-2019-0036323. The gimbal 300 may actively improve the input range of the shooting device 200 by controlling the posture.

[0089] If the gimbal 300 has a rotation axis, the rotation axis may be a yaw axis Y. The yaw axis allows the camera 200 to interact with objects in the space with minimal rotation.

[0090] If the gimbal 300 has two rotation axes, the rotation axes may be a yaw axis and a pitch axis P. Also, if the gimbal 300 has three rotation axes, the rotation axes may be a yaw axis, a pitch axis, and a roll axis R.

[0091] The gimbal 300 may include a power supply unit 310 as a hardware structural element thereof. The power supply unit 310 is wired or wireless and may receive external power through AC or DC. The power supplied to the power supply unit 310 may be used for the gimbal 300 or the computing device 100. Furthermore, the power supply unit 310 may be used to charge a battery built into the gimbal 300 or a battery built into the computing device.

[0092] Furthermore, the gimbal 300 may include at least one gimbal motor (not shown) 330 as a hardware structural element thereof. The gimbal motor 330 changes the direction of the shooting device 200 or the direction of the computing device 100 in which the shooting device 200 is built through the rotation axis. The gimbal motor 330 may be a direct current (DC) motor, a stepper motor, or a brushless motor, but is not limited thereto. In addition to the gimbal motor 330, the gimbal 300 may also include a gear for converting the motor torque.

[0093] The motor 330 of the gimbal 300 can make the shooting device 200 or the computing device 100 attached to the gimbal face the direction of a specific object. Preferably, its various rotation axes are arranged in parallel with the yaw axis, pitch axis, roll axis and other axes of the shooting device 200 or the computing device 100. However, a person skilled in the technical field to which the present invention belongs should understand that there is no reason to be limited to this.

[0094] The gimbal 300 may further include at least one sensor (not shown) 340 as a hardware structural element thereof. The sensor 340 may perform one or more functions of detecting position, angular position, displacement, angular displacement, speed, angular velocity, acceleration, and angular acceleration on the fixed part of the gimbal 300 or the motor 330. The types of the sensor 340 include acceleration sensors, gyroscope sensors, magnetic sensors such as geomagnetic sensors, Hall sensors, pressure sensors, infrared sensors, proximity sensors, motion sensors, light sensors, image (video) sensors, global positioning system (GPS) sensors, temperature sensors, humidity sensors, air pressure sensors, laser radar (LIDAR) sensors, etc., but are not limited thereto.

[0095] In particular, in the computing device 100 , the sensor 340 that cannot be installed in the computing device 100 due to the weight and volume limitations of the portable computing device can be loaded on the gimbal 300 , which can be used to obtain information around the gimbal 300 .

[0096] Below, refer to Figures 3 to 11d For detailed information, please refer to Figure 2 The specific functions and effects of the present invention are realized by the single structural elements briefly described. Figure 2The structural elements shown are implemented in a computing device, but it should be understood that the computing device 100 used to execute the method of the present invention can also make multiple devices linked to each other. For example, the pan-tilt 300 is an independent computing device, and the pan-tilt 300 is linked to the computing device 100. For example, portable computing devices such as portable terminals can also be linked to each other. In this case, the pan-tilt 300 can also replace at least part of the functions performed by the direct portable terminal 100. That is, ordinary technicians in the technical field to which the present invention belongs can link multiple devices in various ways to execute the method of the present invention.

[0097] Figure 3 To illustrate an exemplary flow chart of an image processing method according to an embodiment of the present invention, Figure 4 FIG. 1 is an exemplary block diagram showing a module for executing each step of an image processing method according to an embodiment of the present invention. Figure 5 The figure is an exemplary block diagram showing a machine learning model required for executing a module of an image processing method according to an embodiment of the present invention.

[0098] First, refer to Figure 3 The image processing method of the present invention includes an image acquisition step S1000, in which the image input module 4100 implemented by the computing device 100 acquires the overall image from the shooting device 200 linked by the computing device 100 or the communication unit 110 of the computing device 100.

[0099] Here, the “overall image” refers to an image that contrasts with an image that is a part of the overall image, for example, an object image.

[0100] Next, the image processing method further includes a category classification step S2000, in which the object analysis module 4200 implemented by the computing device 100 detects one or more objects displayed in the overall image, and calculates categories for each of the detected objects to perform classification.

[0101] The category of the object refers to the result of classifying the object into a person, a tree, a dog, etc.

[0102] In the category classification step S2000, the positions of more than one object in the overall image may be calculated as part of the two-dimensional measurement while performing the classification.

[0103] The difference between the two-dimensional measurement and the three-dimensional measurement described below is that the two-dimensional measurement is a measurement based on the two-dimensional coordinate system displayed in the image without considering the three-dimensional depth information, while the three-dimensional measurement is a measurement that considers the two-dimensional coordinates and depth information in the image.

[0104] Specifically, the category classification step S2000 may include: step S2100, adjusting the resolution (resizing) of the overall image to be lower than the initial resolution of the overall image; and step S2200, inputting the resized image into the object classification model M420 to respectively calculate the category, position and importance of the object.

[0105] For example, the processor 120 of the computing device 100 may use the object analysis model M420 to obtain analysis information corresponding to the object included in the image from the image. The analysis information may include at least one of classification information indicating the object category, location information indicating the location of the object, and / or importance information indicating the priority order of the object in the image.

[0106] The object analysis model M420 is a model for performing analysis on objects included in an image, and may include an object classification model, a localization model, an object detection model, a segmentation model, and the like.

[0107] According to an embodiment of the present invention, the processor 120 may distinguish the class of an object in an input image by using an object analysis model M420 for performing object classification. For example, when a person exists in the input image, the processor 120 may obtain an output of "the class of the input image is a person" from the input image by using the object analysis model M420. The above contents are merely examples, and the present invention is not limited thereto.

[0108] According to an embodiment of the present invention, the processor 120 may also output position information indicating the position of an object in an image through the object analysis model M420 for performing classification and localization. For example, when the processor 120 uses the object analysis model M420 for performing classification and localization, a bounding box may be used to identify an object in an image and output position information. The bounding box may return the position information of the object by outputting the left coordinates, right coordinates, upper coordinates, and lower coordinates of the box. The above content is only an example, and the present invention is not limited thereto.

[0109] According to an embodiment of the present invention, the processor 120 may detect at least one object by using the object analysis model M420 for detecting objects. The processor 120 may detect multiple objects and extract location information by performing localization while classifying at least one object using the object analysis model M420 for detecting objects. The above contents are only examples, and the present invention is not limited thereto.

[0110] According to an embodiment of the present invention, the processor 120 may classify pixels through the object analysis model M420 for performing segmentation to distinguish the boundary line and background of the object in the image and detect the object. The above contents are only examples, and the present invention is not limited thereto.

[0111] It should be understood by those skilled in the art that the resizing in step S2100 refers to increasing the processing speed of the object analysis module 4200 by reducing the amount of computation of the object analysis model M420.

[0112] In the result of step S2200, the importance refers to a scale that can be used to determine the priority order of the objects. A high priority order object can be given a high importance, or conversely, a low priority order object can be given a low importance. The content of the priority order will be described in detail later.

[0113] For the object whose importance is above the specified threshold, the computing device 100 may extract and sample the object image as the image of the object from the overall image (crop feed step; S2500). In step S2200, part of the data of the overall image is lost due to the use of the resized image, so the crop feed step S2500 is used to use the image information before the loss for the object with higher importance.

[0114] Continue to refer to Figure 3 The image processing method of the present invention further includes a detailed classification step S3000, in which a detailed classification module 4300 implemented by the computing device 100 generates detailed classification information including the characteristics and status of the object as analysis results of the detection object.

[0115] The object is a concept that includes a spatial object. A spatial object refers to an object corresponding to the “space” itself whose overall image is captured. The detailed classification information of the spatial object may include information of the corresponding space.

[0116] Object properties and states

[0117] The property of an object refers to a property of the object that does not change over time, while the state of an object refers to a property of the object that changes over time.

[0118] Specifically, the characteristics of the object may include information of partial objects that constitute a part of the object or are structural elements of the object. For example, in the overall image, if the detected object is a person, the arms, legs, eyes, and other parts of the person, and the clothes and shoes worn by the person are partial objects of the object.

[0119] In order to detect the partial object, the detailed classification step S3000 may include: step S3920, detecting the partial object that is a part of the object or a structural element of the object; and step S3940, if the partial object is detected, additionally generating analysis results of the characteristics and status of the partial object as part of the detailed classification information.

[0120] Furthermore, the characteristics of the object may include at least one of the main color of the object, an information source body referring to the partial object of the object, an information body referring to the other object when the object is a partial object of the other object, one or more materials of the object including the size of the object, the main material of the object, the transparency of the object, text displayed on the surface of the object, and whether the object can move by itself.

[0121] The size of the object refers to the size measured by two-dimensional measurement or three-dimensional measurement. And the transparency of the object refers to the property that the object can have when it is an object with transparent parts such as a glass window. For example, an opaque object has a value of 0, and a transparent material object such as glass can have a positive value.

[0122] On the other hand, the state of the object may include at least one of the position of the object, the posture of the object, the action of the object, the direction of the object, whether the object is in contact with the ground, and the speed of the object.

[0123] The position of the object refers to a position measured by two-dimensional measurement or three-dimensional measurement. The posture of the object can be inferred by the position information of a part of the object, and the action of the object can be inferred by the posture that is continuous in time.

[0124] Furthermore, the direction of the object may be inferred through the location information of the object or the movement of the object.

[0125] Whether the object is in contact with the ground refers to whether the object is in contact with the ground plane of the space object to which the object belongs. As an example, a chair, a desk, a telephone pole, a car tire, etc. have true values.

[0126] In addition to the characteristics and status, the detailed classification information of the object may also include attributes, which refer to information related to the system input and output of the object. The attributes of the object may include at least one of the raw data of the object, the data input time including the initial input time, and the operation authority granted to the object by the system of the present invention.

[0127] As described above, in order to generate detailed classification information of the object, the detailed classification step S3000 may include: step S3200, selecting a detailed classification model M430 for an individual object, the individual object refers to an object whose importance is above a specified second numerical value, and the detailed classification model M430 refers to a model set composed of at least one pre-trained model in a manner suitable for the individual category in order to obtain the characteristics and the state of each object corresponding to the individual category of the individual object respectively; and step S3400, inputting an individual object image as an image of the individual object into the selected detailed classification model M430 to generate an identifier of the individual object and an object record including the detailed classification information as an object record belonging to the individual object through the identifier.

[0128] The detailed classification model M430 is used to distinguish more than one object. In other words, as detailed classification information generated by the detailed classification model M430, each object may be assigned an identifier that can distinguish each object.

[0129] Furthermore, the object record used herein refers to a record of information including an identifier belonging to each object. As an example of information belonging to an identifier of each object, if the object is a person, the information may include the person's face shape, height, gait, tattoos, hairstyle, etc., and if the object is a dog, the information may include the dog's head shape, hair form and color, breed, etc. The object record may include the belongings of each object, that is, information of another object, which may be an identifier of another object.

[0130] In particular, human gait can be classified through artificial intelligence methodology, for example, as shown in the paper Babaee, M., Li, L., & Rigoll, G. (2019). Person identification from partial gaitcycle using fully convolutional neural networks. Neurocomputing, 338, 116-125. which is a non-technical literature.

[0131] Furthermore, people's hairstyles can be classified using artificial intelligence methodology, for example, as shown in the paper Muhammad, UR, Svanera, M., Leonardi, R., & Benini, S. (2018). Hair detection, segmentation, and hairstyle classification in the wild. Image and Vision Computing, 71, 25-37. https: / / doi.org / 10.1016 / j.imavis.2018.02.001, which is a non-technical literature.

[0132] However, it should be understood by those skilled in the art to which the present invention relates that information other than that shown in the prior art documents may be obtained through artificial intelligence methodology.

[0133] The step S3200 of selecting detailed classification models M430 for individual objects may be performed by the classification model selection module 4320 implemented by the computing device 100. After obtaining the category of the object, the classification model selection module 4320 may select a detailed classification model suitable for the category and an algorithm suitable for the detailed classification model.

[0134] For example, if the category of the object is human, the classification model selection module 4320 can select a detailed classification model to generate detailed classification information such as the person's facial shape, height, gait, tattoo, hairstyle, etc. that can specify the person. If the category of the object is dog, a detailed classification model can be selected to generate detailed classification information such as the dog's head shape, hair shape and color, breed, etc. that can specify the dog.

[0135] In addition to the facial shape of a person, the head shape of a dog can also be classified using artificial intelligence methodology, for example, as described in the paper Mougeot, G., Li, D., & Jia, S. (2019). A Deep Learning Approach for Dog Face Verification and Recognition. In PRICAI 2019: Trends in Artificial Intelligence (pp. 418-430). Springer International Publishing. https: / / doi.org / 10.1007 / 978-3-030-29894-4_34, which is non-technical literature.

[0136] In addition, for the artificial intelligence methodology for classifying dog breeds, please refer to the paper Raduly, Z., Sulyok, C., Vadaszi, Z., & Zolde, A. (2018). Dog Breed Identification Using Deep Learning. In 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics (SISY). IEEE. https: / / doi.org / 10.1109 / sisy.2018.8524715 as a non-technical literature.

[0137] On the other hand, the detailed classification model M430, as a collection of models, may include a measurement model M431 that performs at least one of two-dimensional measurement and three-dimensional measurement on the object to calculate at least one (one or more) of the position of the object, whether the object touches the ground, the direction of the object, the speed of the object, the posture of the object and / or the size of the object. The size of the object may include at least one (one or more) of one-dimensional size, two-dimensional size and / or three-dimensional size, the one-dimensional size includes at least one (one or more) of height, width and / or depth, the two-dimensional size includes the surface area of ​​the object, and the three-dimensional size includes the volume of the object.

[0138] For example, the volume of the object may be calculated based on at least one (or more) of object segmentation, pose and / or depth information of the object.

[0139] If the object is a spatial object, the measurement model M431 may calculate at least one of a system position that becomes an origin in the orientation and coordinates of the spatial object, a Manhattan space that is a volume space of objects included in the space including the spatial object, a ground plane detected in the space, a gravity vector applicable to the space, an empty volume space in the space except for the objects included in the space, a partial object of the spatial object, and a square of the space. The space may be an indoor space or an outdoor space.

[0140] The partial object of a spatial object refers to a partial object of the space that constitutes the spatial object, and the partial object is fixed in the space. For example, the partial object of such a spatial object may include a glass window, a door, a wall, a kitchen, a road, a land bridge, etc.

[0141] The system position refers to the position of the system of the present invention that performs the role of the origin of orientation or coordinates in an indoor space or an outdoor space.

[0142] Furthermore, the direction of the space refers to the direction of an object serving as the space object, based on the system of the present invention or a part of the space.

[0143] The depth information in the three-dimensional measurement can be predicted by an artificial intelligence methodology that derives depth information from the image or provided in addition to the image from other sensors 340 such as a lidar sensor, a radar sensor, an ultrasonic sensor, etc.

[0144] Specifically, step S3400 may also include: step S3410, identifying a length reference object in the object or a partial object that constitutes a part of the object or serves as a structural element of the object through the three-dimensional measurement and determining the two-dimensional length of the length reference object, wherein the length reference object satisfies the condition that the deviation of at least a part of the length is less than a prescribed reference; and step S3420, detecting the ground plane of the object.

[0145] The purpose of the three-dimensional measurement is to use information of more than one object to grasp the relative position between the system of the present invention and the object and / or the absolute position of the object.

[0146] For example, the height of an adult male as an example of a length reference object can be used as a reference length for other types of objects such as doors, pencils, and cups, which can also be used for distance measurement.

[0147] Furthermore, as another example of a length reference object, if multiple doors are detected in an overall image or space (or spatial object), doors of the same design in a space have similar heights and can therefore be used as length reference objects for determining distances.

[0148] Furthermore, an object whose partial object or partial object has relatively small length deviation can be used as a length reference object. For example, since the lateral length of the human eye has relatively small standard deviation, it can be used as a length reference object.

[0149] In step S3410, when measuring the two-dimensional length of the length reference object, if there is posture information of the length reference object, the corrected length can be calculated by reflecting the tilt caused by the posture of the length reference object. The posture of the object and its calculation will be described in detail later.

[0150] Ground Plane

[0151] In step S3420, the ground plane of the object may be detected in a variety of ways. For example, the ground plane in contact with the object may be detected by the direction of a gravity vector applied to the object.

[0152] Furthermore, the ground plane detection of the object may include a method of utilizing Manhattan space detection, a method of generating and extending a ground plane between the length reference objects when there are more than two length reference objects, and a method of generating and extending a ground plane between the length reference object before the movement and the length moving object after the movement when the length reference object moves.

[0153] Figures 6a to 6d FIG. 1 is an exemplary flow chart showing how the image processing method of the present invention can be used to detect the ground plane of an object.

[0154] Reference Figure 6a In a specific first embodiment, the ground plane detection S3420 of the object begins with step S3422a, detecting at least one size-similar object corresponding to a size-similar category in at least one category, wherein the size-similar category satisfies a condition that the size deviation of at least a part of the objects corresponding to a certain category or a part of the objects of the object is less than a prescribed reference. The size may be one or more of width, height, and depth.

[0155] For example, if the size deviation is less than the specified benchmark, the category that meets the condition may be the category of desks. The height deviation of a certain type of desk is relatively small. If the average height of a certain type of desk is 70cm to 74cm, the standard length of similar objects can be set to 72cm.

[0156] Optionally, the size-similar objects may also be ground reference objects. Ground reference objects refer to objects used to detect ground planes, and at least a portion of the object segmentation thereof generally refers to objects that touch the ground, such as ground reference objects such as desks and chairs.

[0157] After step S3422a, the first embodiment of step S3420 includes step S3424a of detecting a ground contact point as a contact point in contact with the lowermost end of the object segmentation of the similar-sized object.

[0158] Figure 7a An exemplary diagram showing object segmentation obtained by an image processing method according to an embodiment of the present invention.

[0159] A method for obtaining object segmentation is known to a person skilled in the art to which the present invention belongs. For example, a non-technical document may be referred to: Mo, K., Zhu, S., Chang, AX, Yi, L., Tripathi, S., Guibas, LJ, & Su, H. (2019). PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding. In 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE.

[0160] Reference Figure 7a In the illustrated chair object segmentation, the lowermost end of the chair leg corresponding regions 712, 714, 716, and 718 can be considered to be in contact with the ground, and the ground contact point 720 can be detected as a connection point with the lowermost end. As another example, in the door object segmentation, the lowermost end is in contact with the ground, and the ground contact point can be detected as a connection point.

[0161] After step S3424a, the first embodiment of step S3420 includes step S3426a, determining the three-dimensional distance of the objects of similar size based on the two-dimensional length of the objects of similar size. For example, compared with the chair near, at least one (or more) of the two-dimensional length (height), width and / or depth measured for the chair far away is relatively small, which can be used to determine the three-dimensional distance.

[0162] After step S3426a, the first embodiment of step S3420 further includes step S3428a, generating a three-dimensional reference plane 730 by connecting the ground contact points 720, and extending the three-dimensional reference plane to generate a ground plane 740. Figure 7a , a three-dimensional reference plane can be generated by connecting the lower ends of the chair legs, which can be further extended into a ground plane 740.

[0163] Among them, the extension of the three-dimensional reference plane is performed to the boundaries such as the edge of the gravity horizontal object, the starting point of the wall, and the bottom of the gravity horizontal object, and the result is the ground plane. Among them, the gravity horizontal object refers to an object that is usually arranged upright and parallel to the direction of gravity, for example, equivalent to a glass window and a wall. Specifically, the gravity horizontal object refers to an object having lines and / or surfaces arranged in the horizontal direction along the gravity vector (for example, a door, a window, etc.). The gravity horizontal object can be an object with more than one line and / or surface in contact with the ground plane (for example, a door frame, a wall, etc.). The horizontal to vertical ratio of the gravity horizontal object (the value obtained by dividing the horizontal length by the vertical length) can also be less than 1.

[0164] On the other hand, the second embodiment of the ground plane detection S3420 using multiple objects begins with step S3422b, detecting similar objects whose similarity is equal to or greater than a specified value. For example, doors, desks, chairs, etc. of the same design can be similar objects.

[0165] Considering that the image processing method of the present invention can be continuously and repeatedly executed, if the image processing method has been applied to the space where the similar object exists, the similar object can be detected by referring to the position and length of the similar object previously detected in step S3422b.

[0166] After step S3422b, the second embodiment of step S3420 further includes step S3424b of detecting at least one lowermost object as the lowermost object among the plurality of similar objects. Figure 7a Similar to the example, since the lowest ends of the chair legs and the door touch the ground, the chair and the door are the lowest objects.

[0167] After step S3424b, the second embodiment of step S3420 also includes step S3426b, detecting the ground contact point as a contact point with the lowermost end of the object segmentation of the lowermost object, and after step S3426b, further including step S3428b, generating a three-dimensional reference plane by connecting the ground contact point, and extending its three-dimensional reference plane to generate a ground plane.

[0168] As a variation of the second embodiment, step S3420 may include: step S3422b', detecting multiple similar objects with a similarity equal to or greater than a specified value; step S3424b', determining whether one of the multiple similar objects is located at the overall upper end of another object in the multiple similar objects; step S3426b', including objects with different layer grounds among the multiple similar objects in sets of different layer grounds based on whether they are located at the overall upper end; and step S3428b', generating a three-dimensional reference plane for each of the different layer ground sets by connecting ground contact points that are in contact with the lowest ends of object segments of the layer ground set objects that are objects in each of the layer ground sets, extending the three-dimensional reference plane to generate a layer ground plane, and generating the layer ground plane into two or more ground planes.

[0169] Figure 7b The figure is an example diagram for explaining the steps of detecting more than two ground planes by an image processing method according to an embodiment of the present invention.

[0170] First, step S3422b' is the same as step S3422b. Figure 7b To explain the “overall upper end” of step S3424b′ and step S3426b′, it can be assumed that the height h of the object whose similarity is greater than a predetermined threshold is within the error range.

[0171] In the overall image, if the objects are separated from each other along the vertical axis (y axis), it is possible to detect (i) the distance between the objects or the height difference of the ground supporting the objects (ii). When the leftmost lower end of the overall image is regarded as the origin of the coordinate system, if there is a distance between the objects, the object 700b having a lower y axis coordinate value is closer to the camera than the other objects 700a, and therefore, the height h b The height h of the object 700a which is larger and has a relatively higher y-axis coordinate value a According to the height h b The prescribed ratio is reduced.

[0172] That is, it can be determined that objects whose heights change according to the y-axis coordinate values ​​according to a specified ratio (for example, 700b and 700a) and objects that have a difference on the horizontal axis (x-axis coordinate value) and whose y-axis coordinate values ​​only differ by a specified level (for example, 700b and 700d) exist on the same ground layer to generate the corresponding ground layer set.

[0173] If the y-axis coordinate value is greater than the other object 700a and the height h c Not based on the height h of other objects 700a a Reduce the specified ratio or the height h cIf the same or larger object 700c exists in different ground layers, different ground layer sets may be generated.

[0174] The layer ground set may be generated by repeatedly executing the inference process described above.

[0175] Alternatively, the height or area of ​​a three-dimensional reference plane formed by each object may be used instead of the height of each object.

[0176] On the other hand, in the process of detecting the ground plane S3420, the third embodiment using the Manhattan space starts from step S3422c, and detects at least one length-similar object corresponding to a length-similar category among objects corresponding to any category or partial objects of its objects in at least one category, wherein the length-similar category refers to a category that satisfies the following condition, that is, the length deviation of at least a part is less than a specified level.

[0177] Among the objects with similar lengths, desks having relatively small height deviations may be included.

[0178] After step S3422c, the third embodiment of step S3420 further includes: step S3424c, detecting a Manhattan space generated based on the set of objects of similar length; step S3426c, detecting the ground of the Manhattan space; and step S3428c, extending the ground of the Manhattan space in a horizontal direction to generate a ground plane.

[0179] More specifically, the detection S3424c of the Manhattan space may include: a first step S3424c-1, detecting a ground object as an object constituting the ground among the objects and generating a boundary of the ground object; a second step S3424c-2, detecting a wall object as an object perpendicular to the boundary of the ground object; and a third step S3424c-3, detecting a ceiling object as an object perpendicular to the wall object as an object other than the ground object.

[0180] That is, the Manhattan space described here refers to a space surrounded by ground objects, wall objects, and ceiling objects. For example, wall objects perpendicular to the ground objects include glass windows, doors, and the like.

[0181] Different from the above-mentioned embodiment, during the detection of the ground plane S3420, the fourth embodiment using the same pattern object starts from step S3422d to detect multiple objects having the same pattern, that is, the same pattern objects.

[0182] After step S3422d, the fourth embodiment of step S3420 also includes step S3426d, detecting the lower end of the same pattern object if the same pattern object is detected; and step S3426d, determining the relative distance between the same pattern objects based on one of the occlusion between the same pattern objects and the length difference in the image between the same pattern objects.

[0183] For example, in step S3426d, if object segmentation distinguishes one object from another object, it can be detected that one object is closer to the system than the other object. Also, in the same pattern object, if one object is smaller than the other object, it can be detected that one object is farther away from the system than the other object.

[0184] After step S3426d, the fourth embodiment of step 3420 further includes step S3428d, generating a virtual plane based on the ground contact points of two or more objects in the same pattern object or three or more points included in one object in the same pattern object. It should be understood by those skilled in the art that if multiple virtual planes are generated, layer-specific virtual planes that regard multiple virtual planes as layers can be generated based on differences in length, state, and position.

[0185] After step S3428d, the fourth embodiment of step S3420 further includes step S3429d of expanding the virtual plane to generate a ground plane.

[0186] Next, the three-dimensional measurement in step S3400 is described again, and the three-dimensional measurement also includes: step S3430, after detecting the ground plane S3420, setting a virtual length reference line on the detected ground plane; and step S3440, determining the system position that becomes at least one origin in the orientation and coordinates of the spatial object and the position of the object based on the distance between the objects or the system position.

[0187] On the other hand, the two-dimensional measurement in step S3400 may include calculating the two-dimensional posture and two-dimensional posture direction measurement of the object based on the relative positions between the partial objects of the object, calculating the two-dimensional width measurement of the object, etc., which can be performed by the detailed classification module 4300 using the measurement model M431 of the detailed classification model M430.

[0188] In terms of the nature of the two-dimensional measurement, the width described herein refers to the width in the whole image or object image without considering the depth. The measurement of the two-dimensional width can be performed based on the classification and measurement of the object segmentation.

[0189] Furthermore, in step S3400, in order to generate the characteristics of the object, the detailed classification model M430 may also include a depth characteristic model M432, which is used to calculate at least one of information of a partial object that constitutes a part of the object or serves as a structural element of the object, depth classification information of the object, the main color of the object, the source of the object, the main body of the object, one or more materials of the object, the transparency of the object, and whether the object can move on its own.

[0190] The depth classification information of the object refers to information for distinguishing the category of the object in detail. For example, if the category of the object is dog, the depth classification information may be the breed of the dog.

[0191] Posture and behavior

[0192] In addition, the detailed classification model M430 may also include a posture judgment model M433, which calculates the posture of the object based on the processing results of the measurement model M431 and the depth characteristic model M432, and may also include a behavior judgment model M434, which classifies the behavior of the object based on the time-continuous posture calculated by the posture judgment model M433.

[0193] For example, the processor 120 of the computing device 100 may obtain posture information of the object from analysis information corresponding to the object using the posture judgment model M433.

[0194] Furthermore, the processor 120 of the computing device 100 may use the behavior judgment model M434 to obtain the behavior information of the object from the posture information of the object.

[0195] The concept of action includes both actions that do not reflect context and behaviors that reflect context. The content of the action will be explained later.

[0196] Since the posture judgment method is different according to the category of the object, the posture judgment model M433 can be a different category posture judgment model for each category. For example, a dog posture judgment model that calculates the dog posture when the dog is sitting and a human posture judgment model that calculates the human posture when the human is sitting can be different from each other. Therefore, the posture judgment model M433 can determine a posture judgment method suitable for the object from a plurality of posture judgment methods based on the analysis information corresponding to the object. The posture judgment model M433 can apply different posture judgment methods based on the classification information representing the category of the object.

[0197] Similarly, since the behavior judgment method varies according to the object category, the behavior judgment model M434 may be a different category behavior judgment model for each category. The behavior judgment model M434 may apply different behavior judgment methods based on classification information indicating the object category.

[0198] The posture determination model M433 may generate posture information of the object based on the analysis information, including information of N-dimensional (eg, one-dimensional, two-dimensional, three-dimensional, etc.) postures corresponding to the object. The posture information of the object may include time-continuous posture information of the object.

[0199] The behavior judgment model M434 can obtain the behavior information of the object based on the posture information of the object, including the N-dimensional (e.g., one-dimensional, two-dimensional, three-dimensional, etc.) movement vector corresponding to the object. Furthermore, the behavior judgment model M434 can generate object behavior information of the object condition based on the posture information of the object, including the N-dimensional (e.g., one-dimensional, two-dimensional, three-dimensional, etc.) behavior classification corresponding to the object. The context of the object may include at least one of the object state and / or the behavior purpose. The behavior classification identifies the position of the partial object included in the object, determines the posture of the object based on the position of the partial object, and can determine and calculate the behavior classification of the object based on the posture of the object. The partial object may include a part of the object or at least one of the structural elements of the object.

[0200] The determination of posture and behavior may be performed in a two-dimensional manner or in a three-dimensional manner. Therefore, the movement vector may include a two-dimensional movement vector including at least one of a two-dimensional direction and / or a speed of the object. Furthermore, the movement vector may include a three-dimensional movement vector of the object, calculated based on at least one of a position, a speed and / or an acceleration of the image captured by the camera.

[0201] According to an embodiment of judging posture and behavior in a two-dimensional manner, in step S3400, the detailed classification module 4300 calculates the two-dimensional motion information of the object based on the time-continuous object image, including the two-dimensional movement vector of the object and the two-dimensional behavior classification of the object. The two-dimensional behavior classification can be calculated by executing the following steps: step S3450a, identifying the positions of the partial objects included in the object; step S3460a, using the posture judgment model M433 to determine the two-dimensional posture of the object based on the relative positions of the partial objects; and step S3470a, using the behavior judgment model M434 to determine the two-dimensional behavior classification of the object based on the time-continuous two-dimensional posture of the object.

[0202] The two-dimensional movement vector refers to the two-dimensional direction and speed of the object, and the two-dimensional behavior classification refers to the behavior type based on the two-dimensional posture judgment of the object.

[0203] On the other hand, according to an embodiment of judging posture and behavior in a three-dimensional manner, in step S3400, the detailed classification module 4300 calculates the three-dimensional motion information of the object based on the time-continuous object image, including the three-dimensional movement vector of the object and the three-dimensional behavior classification of the object. The three-dimensional behavior classification can be calculated by executing the following steps: step S3450b, identifying the positions of the partial objects included in the object; step S3460b, using the posture judgment model M433 to determine the three-dimensional posture of the object based on the relative positions of the partial objects; and step S3470b, using the behavior judgment model M434 to determine the three-dimensional behavior classification of the object based on the time-continuous three-dimensional posture of the object.

[0204] The judgment of two-dimensional posture and behavior is different. Therefore, when judging the behavior of three-dimensional posture, it is necessary to reflect the movement of the shooting device along the depth direction. The computing device 100 uses the linked sensor 340 to calculate or estimate at least one (more than one) of the position, speed and / or acceleration of the shooting device, which can be reflected to calculate the three-dimensional movement vector of the object.

[0205] For example, the system of the present invention can determine the axis of the gimbal motor 330 to rotate the gimbal, based on which the system tracks an object moving in the right direction, i.e., a bicycle. When calculating the three-dimensional movement vector of the bicycle, the three-dimensional movement vector can reflect both the movement vector caused by the motor 330 and the movement vector in the image.

[0206] In order to calculate the three-dimensional position and three-dimensional motion vector of the object, the standard focal distance of the camera 200 and / or the standard focal distance of the whole image or the object image may be used. The relationship between the distance between the camera 200 and the object, the standard focal distance, the three-dimensional height (actual height) of the object, the height of the image (i.e., the vertical size of the image), the height of the object in the image, and the height of the camera 200 is as shown in the following mathematical formula 1.

[0207] Mathematical formula 1

[0208]

[0209] The distance between the photographing device 200 and the object or the three-dimensional height of the object may be calculated using Math Formula 1.

[0210] For example, when measuring the distance between the camera 200 and the object, a distance reference line as a reference may be a distance reference line set using concentric spheres based on the system of the present invention when detecting two or more objects of different lengths. This line is composed of a reference plane circle and a measurement plane circle described later. The steps for setting it are as follows.

[0211] Figure 7cThe figure is a conceptual diagram for explaining the method of generating and using a reference plane circle and a measurement plane circle in an image processing method according to an embodiment of the present invention.

[0212] First, refer to Figure 7c , a reference plane circle 750 is generated along the vertical plane direction of gravity. At this time, when judging the movement of gravity, at least one (or more) of the accelerometer, the gyroscope and / or the gravity horizontal object can be used.

[0213] The reference plane circle 750 is equivalent to the system, that is, the surface of the concentric sphere 740 with the photographing device 200 as the origin, that is, the cross-sectional shape of the concentric sphere surface cut along the xy plane.

[0214] Next, objects with the same length are detected as objects with the same length among objects with similar lengths.

[0215] If you will Figure 7c The set of points separated by an angle θ on the z-axis on the concentric spheres is called a measuring plane circle, and the difference in the angle θ coordinate values ​​between the objects can be determined by utilizing the insertion of the objects of the same length between the measuring plane circles at each angle θ.

[0216] Therefore, the distance between the camera 200 and each object can be determined using the angular coordinate value difference. In this case, at least one (or more) optical characteristics of the standard focal distance, lens and / or aperture size of other camera devices can be auxiliary applied.

[0217] On the other hand, including the behavior judgment model M434, the model disclosed in the present invention can be generated by supervised learning or reinforcement learning. The analysis value (text or classification index) and image information corresponding to each object behavior can be used as training data for supervised learning. In addition, it can also be used for reinforcement learning. As the behavior information of the behavior judgment model M434 is output to the user, if the user gives positive or negative feedback, it can be used to modify the model.

[0218] context

[0219] The detailed classification model M430 as a set may also include a context model M435 for inferring context from the overall image. Context refers to the state of an object and the purpose of an action. For example, if an image of a person peeling an orange with a knife in a kitchen is received, the context model M435 may use the text "a person peeling an orange with a knife in a kitchen" or its corresponding information as context output.

[0220] Action refers to the behavior of an object that does not reflect the context. For example, a person running on a basketball court and a person running on a treadmill both show the action of "running". This behavior is combined with the context described below to determine the action that takes the context into consideration. Therefore, the former is equivalent to the behavior of "playing basketball" and the latter is equivalent to the behavior of "using a treadmill".

[0221] Specifically, the context may be a context interaction object, including at least one of an action and a state of each individual object displayed in the overall image and other objects that interact with the individual object based on the action detection.

[0222] Furthermore, the context may be a spatial context, including a spatial object displayed in the overall image and a place as a type of the spatial object inferred based on individual objects other than the spatial object, at least one of the behaviors and states of each of the individual objects, a person who acts as an individual object equivalent to the subject of the behavior, and a context interaction object of an object that interacts with the individual object through the behavior.

[0223] In one embodiment, the synthesis of actions and contexts for determining behaviors considering contexts can be achieved by supervised learning using an artificial neural network model. For example, continuous images, i.e., a scene of a video, can be regarded as input data, and the artificial neural network model can be trained using training data that labels language analysis (i.e., data represented by language) including actions of objects and spatial contexts as correct answer output data.

[0224] Related non-technical literature includes the paper Wu, Z., Yao, T., Fu, Y., & Jiang, Y.-G. (2017). Deeplearning for video classification and captioning. InFrontiers of Multimedia Research (pp. 3-29). ACM.

[0225] On the other hand, a more detailed method for determining the object behavior of each object can be referred to Wu, C.-Y., Girshick, R., He, K., Feichtenhofer, C., & Krahenbuhl, P. (2020). A Multigrid Method for Efficiently Training Video Models. In 2020 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR). IEEE. https: / / doi.org / 10.1109 / cvpr42600.2020.00023.

[0226] On the other hand, there may be a situation where the action is the same as the spatial context, in which case it is possible that the analyzed behavior does not match the context. For example, a person running in a coffee shop can be analyzed as a "hurried movement" behavior rather than a "running" behavior. This mismatch is because the behavior does not fully reflect the context or the context does not fully reflect the behavior. That is, the behavior can be inferred based on the context, and the context can also be inferred based on the behavior. Therefore, it may be necessary to perform a loop processing of inferring the behavior again based on the context. For example, this can be achieved by using a recurrent neural network (RNN) or a neural network model such as a bidirectional LSTM disclosed in the paper Ul lah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, SW (2018). Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features. IEEE Access, 6, 1155-1166. https: / / doi.org / 10.1109 / access.2017.2778011 as non-technical literature.

[0227] Priority

[0228] The following describes the priority order of objects that is closely related to the importance of the objects. The importance refers to a value used to assign a priority order. Usually, after calculating the importance of each object, the order determined based on the importance can be used as the priority order.

[0229] The priority of objects may include a priority based on permissions, where the objects are assigned based on their permissions. This assigns a priority based on the rights of the various objects that are classified, for example, for the system of the present invention, if an object that is a specific user has the highest permissions, then a high priority may be assigned to that specific user.

[0230] The permission-based priority order can be calculated by executing the following steps: determining whether the object is a predetermined permission person who can operate the computing device based on analyzing the characteristics of the object using the detailed classification model M430; and if the object is a predetermined permission person, setting the predetermined order of the permission persons as the permission-based priority order of the object.

[0231] For example, if the authorized person is expected to be a human, human objects may be detected first, and priority based on authority may be set only for objects detected as humans.

[0232] Furthermore, the computing device of the present invention may use the deep classification information to specify an individual among a plurality of persons, and in order to specify the individual, characteristics of an object or characteristics of a part of the object or an object record of the object may be used.

[0233] On the other hand, the priority of the objects may be a priority based on classification, and a set of objects distinguished based on at least one characteristic including at least the category of the object or the depth classification information of the object is respectively specified in the characteristics of the objects.

[0234] The priority order based on classification can be a priority order given to categories or deep classification information such as people, animals, other objects, etc. according to a preset, and can be manually set by the authorized personnel of the system. As another example, the computing device of the present invention can automatically and variably set the priority order of category or deep classification information based on the pattern of the system of the present invention by the authorized personnel of the system.

[0235] Next, the priority order of the objects may be a priority order based on size, and a set of objects differentiated based on at least one characteristic including at least the size of the object is respectively specified in the characteristics of the objects.

[0236] The size-based priority can be calculated by performing the following steps: the detailed classification module 4300 obtains the object segmentation of the object; screens the objects whose size ratio of the object segmentation in the overall image is equal to or greater than a specified ratio; and sets the size-based priority based on the size of the object segmentation of the screened object. The larger the object is or the smaller the distance between the system of the present invention and the object is, the larger the ratio of the object segmentation of the overall image is, and therefore, it can be regarded as more important and set a high priority.

[0237] Furthermore, the priority of the objects may be a priority based on behavior, and a set of objects differentiated based on at least one state including at least the behavior of the object is respectively specified in the state of the object.

[0238] Specifically, the detailed classification module 4300 may detect an object that performs a specific behavior and assign a priority corresponding to its behavior.

[0239] For example, when a person is detected to be fainting, the highest priority can be set for the fainted person. As another example, when there is a person in the air who issues a command by a specific gesture such as swipe, the object issuing the command can be given a high priority.

[0240] As a final example, the priority order of the objects may be a context-based priority order, where a set of objects differentiated based on at least one state of a context including at least the object is respectively specified in the state of the object.

[0241] According to the context-based priority, the detailed classification module 4300 can perform state analysis or spatial analysis on the overall image to infer object context and spatial context from the overall image, and can set a relatively high priority for the actor or the context interaction object based on whether the object is an actor determined by at least one of the inferred object context and spatial context or a context interaction object that interacts with the actor through the actor's behavior.

[0242] For example, if the spatial context is a context of "people of child age playing baseball on a playground," a relatively high priority may be given to detecting a person holding a baseball bat.

[0243] A person skilled in the art to which the present invention relates should understand that, unless there is a logical contradiction or otherwise specified in the context, the steps and processes described in this specification do not necessarily need to be executed in the order described, and each step and process may be executed simultaneously or at different times (asynchronous times).

[0244] Furthermore, although the steps may be performed only once, preferably, in order to obtain time-continuous images, the steps may be performed in real-time and / or iteratively.

[0245] That is, the image processing method of the present invention may further include step S4000 , returning to the image acquisition step S1000 in order to acquire a new overall image. In this case, the image acquisition step S1000 may be executed again by controlling available resources through the tracking controller 4400 .

[0246] Among them, the tracking controller 440 is a module that is the same as the image input module 4100, the object analysis module 4200 and the detailed classification module 4300. The system of the present invention performs the tracking function of the mobile shooting device, that is, makes the shooting device 200 face the target object or target space, wherein the available resources refer to the hardware and / or software used to implement the tracking.

[0247] In one embodiment, the tracking controller 4400 includes: a target position determination module 4420, used to determine the target position and magnification corresponding to the composition captured by the shooting device 200; and a tracking resource controller 4440, which controls the available resources, i.e., hardware resources and software resources, in order to obtain the overall image of the target position.

[0248] Target location determination module

[0249] Figures 8a to 8c The flowchart is an exemplary flowchart showing the manner in which the image processing method of the present invention can be used to determine the target position.

[0250] exist Figure 8a In one embodiment shown, the target position determination module 4420 may execute: step S4110a, obtaining an object priority order of at least one candidate object displayed in the overall image; step S4120a, obtaining an object segmentation of the candidate object; step S4130a, identifying a context interaction object in the overall image based on the inferred context, the context interaction object being an object detected as interacting with the candidate object; step S4140a, determining that the composition captured by the shooting device 200 includes at least one target object and the context interaction object of the target object, the at least one target object being a candidate object having an object priority order of a specified order or higher than the specified order; and an additional composition prediction step S4150a, expanding the composition based on the direction of the target object to determine the target position and magnification.

[0251] The recognition of contextual interaction objects can be achieved through the context model M435. The content of the context model M435 that can be generated through supervised learning can be referred to the paper Shin, D., & Kim, I. (2018). Deep Image Understanding Using Multilayered Contexts. Mathematical Problems in Engineering, 2018, 1-11. as a non-patent document.

[0252] More specifically, the additional composition prediction step S4150a may include: step S4152a, obtaining the direction of the target object; and step S4154a, obtaining the speed of the target object. As described above, the direction of the target object may be obtained based on the relative positions between the partial objects included in the target object, and the speed of the target object, that is, the three-dimensional motion vector, may be calculated based on the two-dimensional motion vector obtained from the image and the information of the sensor 340.

[0253] Next, the additional composition prediction step S4150a may also include: step S4156a, calculating the influence range of the target object based on the direction and speed of the target object, the influence range of the target object refers to the range within which interaction with the target object can occur; and step S4158a, determining the target position and magnification by reflecting the influence range and expanding the composition.

[0254] Among them, the influence range refers to the spatial range that may be at least temporarily occupied within a specified time range starting from the current time or the spatial range that the context interaction object with interaction possibility may occupy, that is, the spatial range that can physically contact the target object, for example, the range that can physically contact part of a person's body or the range of context interaction objects with which the target object can send and receive signals through eyes, ears, mouth, etc., for example, refers to the field of view (FOV) that can send and receive signals with the target object.

[0255] As an example, in step S4156a, the influence range of the eye is 20m. The influence range of the hand can be 1m. This can be used to adjust the composition in combination with the behavior information (behavior classification) of each object. For example, if "a child standing in a baseball batting box holding a baseball bat" is the target object, the composition can be adjusted in step S4158a to include the child's line of sight and the bat object held by the child. "

[0256] On the other hand, Figure 8b In another illustrated embodiment, the target position determination module 4420 may determine the composition of the shot based on the priority of the object. Specifically, the target position determination module 4420 may execute: step S4120b, obtaining the object priority of at least one candidate object displayed in the overall image; step S4140b, determining at least one target object, wherein the at least one target object refers to a candidate object having an object priority of a specified order or higher than the specified order; and step S4160b, determining the composition shot by the shooting device 200 to include the target object.

[0257] Reference Figure 8cIn another embodiment reflecting the priority of the object, the target position determination module 4420 may execute: step S4120c, obtaining the object priority of at least one candidate object displayed in the overall image; step S4140c, generating the virtual segmentation in a manner that the area of ​​the virtual segmentation is larger as the object priority is higher, which refers to the step of generating the virtual segmentation including the candidate object based on the object priority of the candidate object; step S4160c, determining the parallel state of the attraction, the first repulsion and the second repulsion, the attraction refers to the attraction between the virtual segmentations that increases monotonically with the area of ​​the virtual segmentation calculated based on a prescribed increase function, the first repulsion refers to the first repulsion between the candidate objects that decreases monotonically with the distance between the candidate objects calculated based on a prescribed first decrease function, and the second repulsion refers to the second repulsion between the candidate object and the frame boundary line of the overall image calculated based on a prescribed second decrease function; and step S4180c, in the parallel state, calculating the center location of the at least one candidate object, i.e., the target center, to determine the target position and magnification corresponding to the composition.

[0258] Tracking Resource Controller

[0259] The tracking resource controller 4440 may include: an image conversion controller 4442, which is used to control the image obtained from the shooting device 200 to obtain the new image without controlling the posture of the gimbal 300 and the shooting device 200; and a frame conversion controller 4444, which controls the shooting device 200 to obtain the new image by controlling the gimbal 300 and the shooting device 200 when the gimbal 300 on which the shooting device 200 is installed and includes more than one rotation axis for controlling the posture of the shooting device 200 is linked with the computing device 100.

[0260] The frame conversion controller 4444 and the image conversion controller 4442 can complement each other's operations. The tracking resource controller 4440 preferentially obtains a new image of the desired resolution from the frame conversion controller 4444. If the acquisition fails, the new image can be obtained by controlling the image reconstruction of the image conversion controller 4442.

[0261] In a specific embodiment, the image conversion controller 4442 can execute: step S4310, determining whether there are idle resources for performing image reconstruction; step S4320, if the idle resources exist, loading the pre-acquired entire image or a part of the entire image into the memory as the original image; and step S4330, cropping the original image according to the target position and magnification, or, if the resolution of the original image is less than a predetermined threshold, acquiring the new image based on the target position and magnification by performing super-resolution.

[0262] It should be understood by those skilled in the art that super-resolution can be performed using a neural network such as an autoencoder.

[0263] On the other hand, the frame conversion controller 4444 includes: a gimbal controller 4444a, which controls the direction of the shooting device 200 by operating the one or more rotation axes of the gimbal 300 to achieve the target position (i.e., reach the target position); a zoom controller 4444b, which is used to control the zoom-in and zoom-out of the shooting device 200 to achieve the magnification; and a front and rear shooting device controller 4444c, which is used to control the environmental shooting while reducing the hardware work in the available resources, wherein at least the gimbal controller 4444a is included.

[0264] There may be more than two cameras 200, for example, the cameras 200 may be mounted on the front and back of the portable terminal 100. The front and rear camera controller 4444c may use the front camera (not shown) 200a and the back camera (not shown) 200b to scan the entire space surrounding the system, that is, the surrounding space may be scanned, and the camera 200 may be selected from the front and back cameras 200 according to the user's convenience.

[0265] For example, when the user observes the display of the portable terminal 100, the front and rear camera controller 4444c may select the front camera 200a as a camera for acquiring an image of the user.

[0266] As another example, after the front camera 200a recognizes an object, if a higher resolution image is required for the object than the resolution of the front camera 200a, the front and rear camera controller 4444c can control the back camera 200b with a higher resolution to recognize the object.

[0267] Furthermore, the front and rear camera controller 4444c can utilize the front camera 200a and the rear camera 200b simultaneously or sequentially to minimize the rotation of the axis of the gimbal 300 and acquire images of the surrounding space.

[0268] On the other hand, the frame conversion controller 4444 can control the orientation of the shooting device 200 in one of the portrait and landscape directions according to the aspect ratio of its specific object or the partial object included in its specific partial object (the value of dividing the horizontal length by the vertical length), so that the composition includes at least one specific object or a specific partial object. For example, this can be achieved by rotating the roll axis R of the gimbal 300. In the aspect ratio, the horizontal direction refers to the vertical direction of the gravity vector. In the aspect ratio, the total direction refers to the horizontal direction of the gravity vector. Therefore, the aspect ratio can be the ratio of the horizontal to the vertical direction of a part of a reference plane calculated based on the horizontal length. The corresponding reference plane can be a plane perpendicular to the gravity vector or / and a plane in contact with the ground. However, the corresponding reference plane is not limited to this, and multiple planes can be set as the reference plane. The tendency of the aspect ratio (for example, above 1 or below 1) may not change with the direction of the object.

[0269] Specifically, the frame conversion controller 4444 can control the orientation of the shooting device in one of the longitudinal and lateral directions to prevent the object segmentation of the at least one specific object or the specific partial object or the object box including the at least one specific object or the specific partial object from contacting the frame boundary line of the overall image.

[0270] Figure 9a and Figure 9b The flowchart is an exemplary flowchart showing a method for controlling the orientation of a camera in the image processing method of the present invention.

[0271] Reference Figure 9aFurther describing in detail an embodiment of a frame conversion controller 4444 for orienting the camera device, the frame conversion controller 4444 may execute: step S4210a, if the object segmentation or object frame of at least one specific object or specific partial object makes a first contact with the frame boundary line of the overall image, within the range of the center point of the object segmentation or the object frame in the composition, the gimbal moves in a direction opposite to the first contact position to release the first contact; and step S4220a, after the movement, if the object segmentation or the object frame makes a second contact with the frame boundary line on the opposite side of the first contact position, the orientation of the camera device is converted from one of portrait and landscape to the other by executing (i); and (ii) releasing the second contact by controlling the zoom-in and zoom-out of the camera device.

[0272] Reference Figure 9b Detailed description of another embodiment of the frame conversion controller 4444, the frame conversion controller 4444 may execute: step S4210b, for obtaining a ratio characteristic among the characteristics of the at least one specific object or the specific partial object, the ratio characteristic including a category ratio characteristic, a detailed classification ratio characteristic and a landscape ratio, the category ratio characteristic refers to a landscape ratio (a value obtained by dividing the landscape length by the landscape length) of the specific object or the specific partial object predetermined based on a category or calculated by measuring the specific object or the specific partial object according to a category, the detailed classification ratio refers to a landscape ratio (a value obtained by dividing the landscape length by the landscape length) of the specific object or the specific partial object predetermined based on a detailed classification or calculated by measuring the specific object or the specific partial object according to a detailed classification, the landscape ratio refers to a landscape ratio (a value obtained by dividing the landscape length by the landscape length) of the composition determined by the target position; and step S4220b, determining whether to execute a rolling rotation based on the ratio characteristic and adjusting the orientation of the shooting device according to the execution state, the rolling rotation referring to a work of converting the orientation of the shooting device from one of portrait and landscape to another.

[0273] For example, since the aspect ratio of the TV (the value obtained by dividing the horizontal length by the vertical length) is greater than 1, the orientation of the shooting device 200 can be adjusted to the horizontal direction. The aspect ratio of a standing person (the value obtained by dividing the horizontal length by the vertical length) is less than 1, so the orientation of the shooting device 200 can be adjusted to the vertical direction.

[0274] After step S4220b, the frame conversion controller 4444 of an embodiment of the present invention may further perform: step S4230b, if the object segmentation or object frame of the at least one specific object or specific partial object contacts the frame boundary line of the overall image, then by performing: work (i), converting the orientation of the camera from one of portrait and landscape to the other; and work (ii), releasing the contact by controlling at least one of zoom-in and zoom-out of the camera, so that the at least one specific object or specific partial object is included in the composition. If the object segmentation or object frame does not contact the frame boundary line, then step S4220b is terminated and step S4230b is not performed.

[0275] Optical Character Recognition (OCR)

[0276] Hereinafter, the description of the detailed classification model M430 will be returned to explain its optical character recognition. The classification model M430 may also include an optical character recognition model M436 for reading text displayed on the surface of the object. Therefore, the processor 120 of the computing device 100 may use the optical character recognition model M436 to obtain the characters included in the object from the analysis information corresponding to the object.

[0277] Accordingly, the step S3400 may include a step S3500 of utilizing the optical character recognition model M436 of the detailed classification model M430 to enable the detailed classification module 4300 to perform optical character recognition and calculate partial characteristics of the object as the text of the object.

[0278] The step S3500 is described in detail below. The step S3500 may include a step S3520, where the optical character recognition model M436 first determines whether a character is displayed on the surface of the object.

[0279] In step S3520, deep learning or existing optical character recognition technology can be used to detect characters to determine the object on which the characters are displayed.

[0280] After step S3520, the optical character recognition model M436 may execute optical character recognition execution step S3540, and if characters are displayed on the surface of the object, optical character recognition is performed on the displayed characters (e.g., the entire character and / or a set of characters) to use the result text as the result of its optical character recognition as the characteristic input (storage) of the object.

[0281] According to an embodiment of the present invention, the optical character recognition model M436 may perform the following steps: determining whether a character is displayed on the surface of an object; and when it is determined that the character is displayed on the surface of the object, performing optical character recognition on the entire character including the character.

[0282] According to another embodiment of the present invention, the optical character recognition model M436 may perform the following steps: determine whether characters are displayed on the surface of the object; and when it is determined that the characters are displayed on the surface of the object, perform optical character recognition on each character set.

[0283] How to distinguish character sets

[0284] At least one of the processor 120 or the optical character recognition model M436 may distinguish the character set to which the character belongs based on at least one of the type, shape, size, or configuration of the character. Also, the processor 120 may distinguish a character set with a high probability of being closely related during the process of analyzing a context.

[0285] The method for distinguishing the character set to which a character belongs or the character set to which a context belongs is as follows: For example, at least one of the processor 120 or the optical character recognition module M436 may distinguish the character set based on the character set characteristics.

[0286] The characteristics of a character set may include one or more of the language type, line spacing, character spacing, length (e.g., aspect ratio), size, thickness, color, font (font), style, characters related to the beginning or end of a word, spaces (e.g., spaces between at least one boundary of the next word, object, or part of an object), or characters displayed in the position of an object. Therefore, at least one of the processor 120 or the optical character recognition model M436 may use one or more of the characteristics of the language type, line spacing, character spacing, length (e.g., aspect ratio), size, thickness, color, font (font), style, characters related to the beginning or end of a word, spaces (e.g., spaces between at least one boundary of the next word, object, or part of an object), or characters displayed in the position of an object to distinguish a character set.

[0287] For example, at least one of the processor 120 or the optical character recognition model M436 can distinguish the completed text or context according to the language type of the character. If the recognized characters are in multiple languages, such as the cover of a product manual, at least one of the processor 120 or the optical character recognition model M436 can distinguish the completed text or context according to the language type of the character.

[0288] As another example, for text of the same size and font, at least one of the processor 120 or the optical character recognition model M436 may distinguish paragraphs based on at least one of line spacing, character spacing, length, or color.

[0289] As another example, at least one of the processor 120 or the optical character recognition model M436 can distinguish character sets based on the top or start position of a paragraph. The nature or title of the paragraph can also be displayed in bold style or different sizes at the top or start position of a paragraph.

[0290] Objectification of character sets

[0291] The processor 120 and / or the optical character recognition model M436 may distinguish character sets based on the "method for distinguishing character sets". The processor 120 and / or the optical character recognition model M436 may identify character set objects based on the character sets. A character set object refers to a character set set for an object. A character set object may include at least one of a character image, text information, or a character set characteristic. For example, before optical character recognition or natural language understanding (processing), a character set object may include a character image. As another example, after optical character recognition or natural language understanding (processing), a character set object may include text information. Among them, after optical character recognition or natural language understanding (processing), the character set object may also be separated into multiple character set objects.

[0292] The processor 120 and / or the optical character recognition model M436 may distinguish one or more character sets displayed on the surface of an object or a portion of an object. The processor 120 and / or the optical character recognition model M436 may perform contextual analysis based on at least one of the characteristics or state information of the object or portion of the object displaying the character set. The characteristics and / or state of the object and / or portion of the object may be included in the character set characteristics.

[0293] The processor 120 and / or the optical character recognition model M436 may use the type and / or position of the object or partial object of the displayed character set as additional information for natural language understanding (processing) and / or context analysis.

[0294] For example, when the characters displayed on the shirt (object or part of the object) are the same as the characters displayed on the car (object or part of the object), the processor 120 and / or the optical character recognition model M436 may use the characters (e.g., brand name) displayed on the shirt or the car as additional information. The processor 120 and / or the optical character recognition model M436 determines that the characters displayed on the car can reflect the characteristics of the object and / or the characteristics of the overall context (determined to have a relatively high probability of reflecting), and the characters displayed on the car may be used as additional information.

[0295] When there are multiple characters for an object, the processor 120 and / or the optical character recognition model M436 may obtain the characteristics of the object based on the positions where the multiple characters are displayed. The meanings of the multiple characters are different depending on the positions. For example, when there are characters displayed on the front of a shirt (object or part of an object) and characters displayed on a shirt label (e.g., a care label) at the same time, the processor 120 and / or the optical character recognition model M436 determines that the characters displayed on the shirt label can reflect the characteristics of the object (determined to have a relatively high probability of reflection), and the characters displayed on the shirt label can be used as additional information.

[0296] The processor 120 and / or the optical character recognition model M436 may perform optical character recognition and / or natural language understanding (processing) on ​​the objectified character set according to each character set object. The processor 120 and / or the optical character recognition model M436 may perform optical character recognition and / or natural language understanding (processing) on ​​more than one character set object based on the character set characteristics. Among them, the processor 120 and / or the optical character recognition model M436 may also perform optical character recognition and / or natural language understanding (processing) on ​​more than one character set object sequentially or simultaneously based on the character set characteristics.

[0297] The unit of optical character recognition and / or natural language understanding (processing) is not limited to the entire character on the surface of an object or a part of an object, and a character or a character set classified based on the characteristics of the character set can be used as the minimum unit.

[0298] On the other hand, the optical character recognition execution step S3540 can be executed in a variety of ways.

[0299] For example, step S3540 of performing optical character recognition on characters (e.g., entire characters and / or character sets) executed by the processor 120 of the computing device 100 may include the following steps: extracting at least one image sample from an object displaying characters; determining a boundary line of a character display area from at least one image sample; generating a character display image (e.g., an entire character display image and / or a character set display image) based on at least one of the boundary lines or boundary points within the boundary lines in at least one image sample; and performing optical character recognition on the character display image.

[0300] An image sample refers to an image pattern existing in at least one position of a boundary line and / or a boundary point. The image pattern may include at least one of a partial character, a boundary portion of a character, a part of a character and / or a background. The background refers to an image pattern that does not constitute a character.

[0301] The step of generating a character display image based on at least one of the boundary lines in at least one image sample or the boundary points within the boundary lines and the step of performing optical character recognition on the character display image may include the following steps: acquiring an image pattern located at least one of the boundary lines or the boundary points in at least one image sample as a boundary marker; and generating a character display image as a partial image included in the image of an object displaying characters, wherein the character display image is an image including the boundary marker. The boundary marker may include at least one of a start marker and / or an end marker. The start marker refers to an image pattern corresponding to the start pattern of the character. The end marker refers to an image pattern corresponding to the end character of the character.

[0302] The step of acquiring at least one image pattern located at a boundary line or a boundary point in at least one image sample as a boundary marker may include the following steps: using the tracking controller 4400 to acquire a character image including a start mark and a character recognition rate above a threshold; and determining whether the character image includes an end mark.

[0303] Furthermore, when the character image includes an end mark, the processor 120 may determine the character image as a character display image.

[0304] When the character image does not include an end mark, the processor 120 may use the tracking controller 4400 to obtain an additional character image including the next mark of the last mark and having a character recognition rate above a threshold value from the boundary mark included in the character image. The processor 120 may merge the additional character image with the character image to generate a merged character image. Furthermore, when the merged character image includes an end mark, the processor 120 may determine the merged character image as the character display image.

[0305] On the other hand, as another example, step S3540 of performing optical character recognition on characters executed by the processor of the computing device 100 may include the following steps: using a tracking controller to acquire an object image including the starting point of the character and with a character recognition rate above a threshold; performing optical character recognition on an initial text area of ​​the object image; and performing natural language understanding (NLU) on a first text as a first result of the optical character recognition to calculate a first semantic value as a semantic value.

[0306] When the first semantic value is above the threshold, the processor 120 may determine the first text as a result text as a result of optical character recognition.

[0307] When the first semantic value is less than the threshold, the processor 120 may perform optical character recognition on the next text area of ​​the initial text area. The processor 120 performs natural language understanding on the second text as the first result of the optical character recognition to calculate the second semantic value as a semantic value. Moreover, when the second semantic value is above the threshold, the processor 120 may determine the second text as the result text as the result of the optical character recognition.

[0308] The initial character area refers to an area occupied by characters detected as being arranged with the highest priority in the character arrangement method according to the language used by the characters.

[0309] The computing device 100 may be linked with the shooting device 200 and the gimbal 300. For example, the computing device 100 includes the shooting device 200, and may be linked with the gimbal 300 for controlling the posture of the shooting device 200.

[0310] The processor 120 of the computing device 100 may utilize the tracking controller 4400 to operate one or more rotation axes of the gimbal 300 to control the direction of the photographing device 200 .

[0311] The processor 120 may utilize the tracking controller 4400 to acquire an image by zooming in and / or zooming out of the photographing device 200 .

[0312] Figures 10a to 10c To illustrate an exemplary flow chart of the manner in which optical character recognition may be performed in the image processing method of the present invention, Figures 11a to 11d The figure is an example diagram for explaining the execution mode of optical character recognition of the image processing method of the present invention.

[0313] First, refer to Fig.10a and Fig.11a In an embodiment utilizing boundary markers, the optical character recognition execution step S3540a begins with step S3542a of extracting an image sample from an object displaying the character (e.g., a text file of reference numeral 1110), and after step S3542a, includes step S3544a of dividing the boundary line of the character display area (area) from the image sample (e.g., a closed line of reference numeral 1130).

[0314] The image sample refers to an image pattern existing at a boundary line or boundary point, and the image pattern may be composed of a part of a character, a boundary part of a character, a part of a character, or a background. The boundary line refers to the boundary line of a character group displayed on the surface of an object, and the boundary point refers to a set of points that can constitute a boundary line. The background refers to an image pattern that cannot constitute a character by itself.

[0315] The optical character recognition execution step S3540a of one embodiment of the present invention also includes a boundary marker acquisition step S3546a. After step S3544a, the image pattern of the boundary point located on the boundary line or within the boundary line in the image sample is acquired as a boundary marker. In this step, the image pattern in the boundary marker corresponding to the start character in the overall character is used as the start marker, and the image pattern in the boundary marker corresponding to the end character in the overall character can be used as the end marker.

[0316] The boundary marker may be used as a marker for determining a coordinate region, where the coordinate region refers to the space occupied by each character belonging to the overall character in the object image.

[0317] The boundary mark, that is, the boundary part of the character can be used as an image pattern located at the boundary line or a boundary point within the boundary line. The boundary part of the character refers to a part of the image that constitutes a part of the character extending in the horizontal direction or the vertical direction of the character in a certain text. For example, in In Korean, of and of It is a part of a character and can be used as a boundary marker as the boundary line of the character.

[0318] Furthermore, a part (or the whole) of a character may be used as a kind of boundary marker. For example, in the text "ABCDEFG", A and G may be used as boundary markers as a part of a character located at a boundary point and / or a boundary line.

[0319] The acquisition of the start mark or the end mark can reflect the character configuration method of the language based on the text composed of the whole characters. For example, in the process of performing optical character recognition on Korean and English books, as shown in FIG11 , the image sample 1132 displayed on the upper left side of the character display area can be a start mark, and the image sample 1134 on the lower right side of the area can be an end mark.

[0320] More specifically, the boundary mark acquisition step S3546a may include step S3546a-1, controlling available resources through the tracking controller 4400 to acquire an image including the start mark and having a character recognition rate above a specified threshold as a character image.

[0321] The character recognition rate refers to the ratio of a specific character in a character image to be recognized as arbitrary text. For example, although it is possible to infer the existence of an arbitrary character from a long-distance image of an object represented by a displayed character based on deep learning, it is difficult to determine what the character is because the long-distance image is small or has low resolution, which may result in a decrease in the character recognition rate of optical character recognition.

[0322] In other words, the character image with a character recognition rate above the specified threshold refers to a resolution that allows optical character recognition. In the process of executing this step S3546a-1, in order to obtain an enlarged character image with a sufficiently high resolution, the system of the present invention can control the enlargement, reduction and rotation of the camera 200 through the tracking controller 4400. For example, referring to Fig.11a In the composition including both the text file 1110 and the person 1120 , the text file 1110 is located at the center of the image through the rotation control 1160 and the zoom controls 1170 and 1180 of the photographing device 200 .

[0323] After step S3546a-1, an embodiment of the boundary mark acquisition step S3546a includes: an end mark judgment step S3546a-2, judging whether the acquired character image includes the end mark; and executing step S3546a-3 again. If the acquired character image includes the end mark, the character image is determined as the overall character display image. If the acquired character image does not include the end mark, the tracking controller 4400 controls the available resources to search for the next mark of the last mark in the boundary mark displayed by the character image to obtain an additional character image as an image including the next mark and with a character recognition rate above a specified threshold, and merge the additional character image into the character image, thereby starting execution again from the end mark judgment step S3546a-2.

[0324] In step S3546a-3, the control of the available resources may include the control of the pan / tilt platform 300. For example, for Korean and English books, the control of the pan / tilt platform 300 may be an auxiliary control that enables the camera 200 to scan from the left side to the right side of the character display area or from the top to the bottom of the area (for example, referring to Fig.11b Rotation control of reference numeral 1190).

[0325] Furthermore, it should be understood by those skilled in the art that in step S3546a-3, the merging of character images may be performed using a variety of image stitching techniques. Fig.11b Between two or more images captured with a time interval or a space interval, image merging can be achieved by using overlapping image patterns 1136a and 1136b.

[0326] After the boundary mark acquisition step S3546a, the optical character recognition execution step S3540a of an embodiment of the present invention may include step S3468a, performing optical character recognition on the overall character display image to obtain text as a partial image including the object image displaying the characters, wherein the overall character display image refers to an image including both the start mark and the end mark.

[0327] On the other hand, refer to Fig.10b and Fig.11c In the second embodiment of the optical character recognition execution step S3540, the optical character recognition execution step S3540b includes step S3544b, starting with step S3542b of extracting an image sample from the object displaying the character, and after step S3542b, determining the boundary line 1130 of the character display area (area) from the image sample.

[0328] Next, the optical character recognition execution step S3540b includes step S3546b, obtaining at least one image pattern 1142a, 1142b that is not located on the boundary line from the image sample as a segmentation mark, using the segmentation mark to merge the character image and perform optical character recognition on the character display image as the generated image.

[0329] In step S3546b, the segmentation mark refers to the image patterns 1142a, 1142b that overlap each other between two or more images taken at intervals in time or space. The same segmentation mark is captured from different images and used to achieve image merging, that is, image splicing can be achieved.

[0330] Finally, refer to Fig.10c and Fig.11d In the third embodiment of the optical character recognition execution step S3540, the optical character recognition execution step S3540c includes step S3542c, which is used to obtain the object image including the starting point of the overall character and the character recognition rate is above the specified threshold, that is, the object image is controlled by the tracking controller 4400 to control the available resources so that the object image has a specified resolution for optical character recognition.

[0331] As a policy for the tracking controller 4400 to execute step S3542c, in addition to the condition that the object image has a prescribed resolution at which optical characters can be recognized, a condition that the object image includes as many characters as possible at the resolution may be given.

[0332] Next, the optical character recognition execution step S3540c also includes a step S3544c of performing optical character recognition on the initial text area 1152 of the acquired object image.

[0333] In which, considering the common configuration mode of the characters of the language used by the text, the initial text area refers to the area occupied by the first configured text detected.

[0334] For example, if the step S3544c is initially executed, the initial text area may be determined by scanning a text symbol indicating the end point of a text such as "," ".", "?", etc. If such a text symbol is scanned, the area between the start character and the text symbol may be submitted as the initial text area 1152.

[0335] As another example, if step S3544c is initially executed, when the size of the starting character is compared with the size of the next character in sequence, the initial text area 1152 can also be determined by the area occupied by the text composed of characters connected to the starting character before the size of the character becomes above a specified threshold.

[0336] As another example, if step S3544c is initially executed, when the size of the next character space is compared sequentially based on the size of the space between the starting character and the next character, the initial text area 1152 can also be determined by the area occupied by the text composed of characters connected to the starting character before the size of the space becomes above a specified threshold.

[0337] Subsequently, the optical character recognition execution step S3540c also includes a semantic value calculation step S3546c, which performs natural language understanding (NLU) on the first text as the first result of the optical character recognition in step S3544c to calculate a semantic value as the semantic numerical value of the first text.

[0338] Then, the optical character recognition execution step S3540c also includes step S3548c. If the semantic value in step S3546c is above the specified threshold, the first text is determined as the result text and the optical character recognition is completed. If the semantic value is less than the specified threshold, optical character recognition is performed on the next text area 1154 of the initial text area and the semantic value calculation step S3546c is executed again.

[0339] The calculation of semantic values ​​can be achieved cumulatively, because natural language understanding is not only applied to one text for which optical character recognition is achieved, that is, the text for which optical character recognition was previously achieved can be combined with new text and applied again to a larger text area.

[0340] To this end, step S3548c may repeatedly perform optical character recognition on a new text area including the next text after the text symbol existing in the initial text area.

[0341] Of course, even if the optical character recognition is performed to the last word, if its semantic value does not reach the specified threshold, it may be detected as a text without the whole character.

[0342] As described above, by executing the steps of the present invention, a computing device for processing an image of a photographed object can grasp various information of the object detected from the image.

[0343] For the convenience of explanation, although the structural elements shown in the drawings are implemented by a computing device, for example, a portable terminal, in order to execute the method of the present invention, the computing device 100 may also be composed of multiple devices that are linked to each other. Therefore, in addition to the portable terminal, each step of the method of the present invention can be executed by the gimbal 300 with a built-in communication unit and a processor. In addition, it can be directly executed by a computing device or executed by another computing device that is linked to the computing device with the assistance of the computing device.

[0344] As described above, according to all the embodiments and variations, the method and device of the present invention can use images to identify and track one or more objects, and can dynamically obtain information about the objects and the environment. In particular, the state information of the object can be remotely obtained through images, and the object interacting with the object can be judged based on the image. Higher-resolution detailed images of the relevant parts of the object can be remotely obtained, and characters printed on the object or output by other devices such as displays can be remotely grasped. Therefore, it has the advantage of enabling portable computing devices to use images for remote input.

[0345] Based on the above description of multiple embodiments of the present invention, it should be understood by those skilled in the art that the method and / or process of the present invention and its steps can be implemented by hardware, software or any combination of hardware and software applicable to a specific embodiment. The hardware may include a general-purpose computer and / or a special-purpose computing device or a specific computing device or a specific state or structural element of a specific computing device. The process can be implemented by one or more processors with internal memory and / or external memory, such as a microprocessor, a controller, such as a microcontroller, an embedded microcontroller, a microcomputer, an arithmetic logic unit (ALU), a digital signal processor (digital signal processor), such as a programmable digital signal processor or other programmable device. And, alternatively, the process can be implemented by executing an application specific integrated circuit (ASIC), a programmable gate array (programmable gate array), such as a field programmable gate array (FPGA), a programmable logic unit (PLU) or a programmable array logic (PAL) or other instructions (instruction) and other devices or combinations of devices that can respond. The processing device can execute an operating system (OS) and one or more application software executed on the operating system. In addition, the processing device can also access, store, operate, process and generate data in response to the execution of the software. For ease of understanding, although only the case of using one processing device is described, it should be understood by ordinary technicians in the technical field to which the present invention belongs that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, other processing configurations such as parallel processors may also be included.

[0346] Software may include a computer program, code, instruction, or a combination thereof, and may be configured as desired or may be used to instruct a processing device, either independently or in combination. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, in order to be analyzed by a processing device or to provide quality or data to a processing device. Software may be distributed across computer systems connected via a network, or may be stored or executed by a distributed method. Software and data may be stored in one or more machine-readable recording media.

[0347] Furthermore, the object of the technical solution of the present invention or the part that contributes to the prior art can be implemented by program instructions that can be executed by a variety of computer structural elements and recorded in a machine-readable medium. The machine-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the machine-readable recording medium can be known instructions that are specially designed for the embodiment or can be used by ordinary technicians in the field of computer software. For example, the machine-readable recording medium may include magnetic media such as hard disks, floppy disks and magnetic disks, optical recording media such as CD-ROMs, DVDs, Blu-rays, magneto-optical media such as floptical disks, and read-only memory (ROM), random access memory (RAM), flash memory, etc., and hardware devices specially designed for storing program instructions. For example, program instructions are not only created by any of the described devices, but may also be created using structured programming languages ​​such as C, object-oriented programming languages ​​such as C++, or high-level or low-level programming languages ​​(assembly languages, hardware technology languages, and database programming languages ​​and technologies) for storage, compilation, or interpretation on a processor, processor architecture, or a dual combination of different hardware and software combinations, or a machine that can execute any other program instructions, and include not only machine code, bytecode, but also high-level language code that a computer can execute using an interpreter, etc.

[0348] Therefore, in one embodiment of the present invention, when the methods and combinations thereof described above are performed by more than one computing device, the methods and combinations thereof may be implemented by code capable of executing each step. In another embodiment, the methods may be implemented by a system that performs the steps, multiple methods may be distributed in multiple ways by multiple devices or all functions may be integrated in a dedicated device, independent device or other hardware. In another embodiment, the device for performing the steps associated with the process may include any hardware and / or software described above. All such sequential combinations and combinations are within the scope of the present invention.

[0349] For example, in order to perform the work of the embodiment, the hardware device may work as one or more software modules, and vice versa. The hardware device is combined with a memory such as ROM / RAM for storing program instructions, and may include a processor such as an MPU, CPU, GPU, TPU to execute instructions stored in the memory, and may include a communication unit capable of sending and receiving signals with an external device. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving instructions created by developers.

[0350] Although the present invention has been described above through specific matters such as specific structural elements and limited embodiments and drawings, this is only for further understanding of the present invention in detail, and the present invention is not limited to the embodiments, and ordinary technicians in the technical field to which the present invention belongs can make various modifications and deformations based on the described contents.

[0351] Therefore, the concept of the present invention is not limited to the embodiments described above. In addition to the scope of protection claimed by the invention disclosed in this specification, equivalent technical solutions or equivalent substitutes in the scope of protection claimed by the invention belong to the scope of the concept of the present invention. For example, even if the described technology is performed in a different order from the described method and / or the described system, structure, device, circuit and other structural elements are combined or combined in a different implementation from the described method, or even if they are replaced or replaced by other structural elements or equivalent technical solutions, appropriate results can be achieved.

[0352] Its equivalent technical solutions or equivalent substitutes include logically equivalent methods, for example, methods that can produce the same results as executing the method of the present invention. The true meaning and scope of the present invention are not limited to the examples described, and should be understood as the broadest meaning permitted by law.

[0353] Industrial Applicability

[0354] The present invention can be used in a device or system for processing an image acquired from a photographing device.

Claims

1. A method for processing an image executed by a computing device including a processor, the method comprising the following steps: Acquire images; Acquire analysis information corresponding to an object included in the image from the image using an object analysis model; and Using an optical character recognition model, obtaining characters included in the object from analysis information corresponding to the object, The optical character recognition model is used to perform the following steps: Determining whether characters are displayed on the surface of the object; extracting at least one image sample from the object displaying the character; Determining a boundary line of a character display area from the at least one image sample; Acquire, in the at least one image sample, an image pattern of at least one of the boundary lines or boundary points within the boundary line as a boundary marker; generating a character display image as a partial image included in the object image displaying the character, the character display image being an image including the boundary mark; and Optical character recognition is performed on the character display image.

2. The method according to claim 1, wherein: The image sample is an image pattern existing at at least one position of the boundary line or the boundary point. The image pattern includes at least one of a portion of a character, a boundary of the character, a part of the character, or a background.

3. The method according to claim 1, wherein: The boundary mark includes at least one of a start mark or an end mark, the start mark is an image pattern corresponding to the start character of the character, and the end mark is an image pattern corresponding to the end character of the character.

4. The method according to claim 3, wherein: The step of acquiring an image pattern located at least one of the boundary line or the boundary point in the at least one image sample as the boundary marker comprises the following steps: Using a tracking controller to acquire a character image including the start mark and having a character recognition rate above a threshold; and Determine whether the character image includes the end mark.

5. The method according to claim 4, wherein: The method further includes the following steps: if the character image includes the end mark, determining the character image as the character display image.

6. The method according to claim 4, wherein: The following steps are also included: If the character image does not include the end mark, using the tracking controller to obtain an additional character image including the next mark of the last mark and the character recognition rate being above the threshold value from the boundary marks included in the character image; merging the additional character image into the character image to generate a merged character image; and If the combined character image includes the end mark, the combined character image is determined as the character display image.

7. The method according to claim 1, wherein: After the step of determining whether characters are displayed on the surface of the object, the following steps are included: Using a tracking controller, acquiring an image of the object including a starting point of the character and having a character recognition rate above a threshold; performing the optical character recognition on an initial text region of the object image; and Natural language understanding is performed on a first text as a first result of the optical character recognition to calculate a first semantic value as a semantic numerical value.

8. The method according to claim 7, wherein: The method further includes the following step: if the first semantic value is above the threshold, determining the first text as a result text obtained as a result of the optical character recognition.

9. The method according to claim 7, wherein: The following steps are also included: If the first semantic value is less than the threshold, performing the optical character recognition on a text area next to the initial text area; Performing natural language understanding on a second text as a first result of the optical character recognition to calculate a second semantic value as a semantic numerical value; as well as If the second semantic value is greater than the threshold, the second text is determined as a result text resulting from the optical character recognition.

10. The method according to claim 1, wherein: The computing device is linked with the shooting device and the pan-tilt head. The step of acquiring the image comprises the following steps: Using a tracking controller to operate one or more rotation axes of the pan / tilt head to control the direction of the camera; and The image is acquired by utilizing the tracking controller by zooming in or out of the photographing device.

11. A non-transitory computer-readable medium comprising a computer program, wherein: The computer program causes the computing device to execute a method for processing an image, The method comprises the following steps: Acquire images; Acquire analysis information corresponding to an object included in the image from the image using an object analysis model; and Using an optical character recognition model, obtaining characters included in the object from analysis information corresponding to the object, The optical character recognition model is used to perform the following steps: Determining whether characters are displayed on the surface of the object; extracting at least one image sample from the object displaying the character; Determining a boundary line of a character display area from the at least one image sample; Acquire an image pattern of at least one of the boundary points located on the boundary line or within the boundary line in the at least one image sample as a boundary marker; generating a character display image including the boundary mark as a partial image included in the object image displaying the character, the character display image being an image including the boundary mark; and Optical character recognition is performed on the character display image.

12. A computing device comprising: processor; as well as Ministry of Communications, The processor performs the following steps: Acquire images; Acquire analysis information corresponding to an object included in the image from the image using an object analysis model; and Using an optical character recognition model, obtaining characters included in the object from analysis information corresponding to the object, The optical character recognition model is used to perform the following steps: Determining whether characters are displayed on the surface of the object; extracting at least one image sample from the object displaying the character; Determining a boundary line of a character display area from the at least one image sample; Acquire an image pattern of at least one of the boundary points located on the boundary line or within the boundary line in the at least one image sample as a boundary marker; generating a character display image as a partial image included in the object image displaying the character, the character display image being an image including the boundary mark; and Optical character recognition is performed on the character display image.

Images