Robot control device and control method thereof

Abstract

To provide a robot control device for identifying a target object by using a camera and a lidar, and a control method thereof.SOLUTION: A device for controlling a robot according to the present invention includes a lidar, a camera, a memory in which a classifier group including a plurality of classifiers and a neural network model are stored, and a processor, wherein, when a point cloud corresponding to an external object is acquired through the lidar, the processor acquires a virtual object represented in two dimensions by projecting the point cloud onto a designated surface, after a visual object corresponding to a virtual object is identified in an image acquired through a camera, a part of the image including the visual object is input to a neural network model, a specified number of feature maps for the part of the image are acquired from the neural network model, and then the feature maps are input to a classifier group to identify whether an external object corresponding to the visual object is a target object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a robot control device and a control method thereof, and more particularly to a technology for identifying an external object using a camera and a lidar. [Background technology]

[0002] Recently, in the field of robotics, various robotic technologies have been researched, particularly technologies that allow a robot to move while tracking a target.

[0003] When a robot tracks a target and plans a path, it needs to accurately identify the target. To this end, research is being conducted into identifying targets using a variety of sensors.

[0004] Cameras and LIDARs are used as various sensors for identifying targets. Since cameras acquire two-dimensional image data and LIDARs acquire spatial data expressed as three-dimensional data, it is necessary to process the data acquired from the sensors in order to track objects using cameras and LIDARs. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2022-128579 Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above-mentioned conventional technology, and an object of the present invention is to provide a robot control device and a control method thereof for identifying a target object using a camera and a lidar. [Means for solving the problem]

[0007] According to one aspect of the present invention, there is provided a robot control device including a LiDAR (light detection and ranging), a camera, a memory storing a classifier group including a plurality of classifiers and a neural network model, and a processor. The processor is configured to acquire a point cloud corresponding to an external object via the LiDAR, project the point cloud onto a designated plane to acquire a virtual object represented in two dimensions, identify a visual object corresponding to the virtual object in an image acquired via the camera, input a portion of the image including the visual object to the neural network model, acquire a designated number of feature maps for the portion of the image from the neural network model, and input the feature maps to the classifier group to identify whether the external object corresponding to the visual object is a target object.

[0008] The processor may be configured to identify whether the external object is the target object based on a region of interest (ROI) of each of the feature maps input to the classifier group. The processor may be configured to identify whether the external object is the target object by identifying pixel values ​​of the region of interest using each of the plurality of classifiers. The processor may be configured to identify whether the external object is the target object by inputting a sum of pixel values ​​of the region of interest into a first Gaussian probability distribution and a second Gaussian probability distribution. The processor may be configured to identify whether the external object is the target object based on a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution. The processor may be configured to identify the external object as the target object if a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is greater than or equal to a first critical value. The processor may be configured to suspend judgment on the external object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a first critical value and greater than or equal to a second critical value smaller than the first critical value. The processor may be configured to identify that the external object is not the target object if a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a second critical value that is smaller than a first critical value. The processor may be configured to train the plurality of classifiers using a first training feature map for tracked targets and a second training feature map for generic objects. The processor may be configured to obtain the feature map by propagating a portion of the image through a plurality of convolution layers included in the neural network model. The processor may be configured to initialize at least one of the partial classifiers excluding the representative classifier when the representative classifier is selected from the plurality of classifiers. The processor may be configured to assign a region of interest based on at least one of the feature maps while initializing at least one of the plurality of classifiers.

[0009] In order to achieve the above object, according to one aspect of the present invention, a robot control method executed by a processor includes the steps of: acquiring a point cloud corresponding to an external object through a light detection and ranging (LiDAR) by the processor, and projecting the point cloud onto a designated plane to acquire a virtual object represented in two dimensions; identifying a visual object corresponding to the virtual object in an image acquired through a camera, and inputting a portion of the image including the visual object into a neural network model stored in a memory; and acquiring a designated number of feature maps for the portion of the image from the neural network model, and inputting the feature maps into a classifier group stored in the memory to identify whether the external object corresponding to the visual object is a target object.

[0010] The robot control method may include identifying whether the external object is the target object based on each region of interest (ROI) of the feature map input to the classifier group. The robot control method may include identifying whether the external object is the target object by identifying pixel values ​​of the region of interest using each of a plurality of classifiers included in the classifier group. The robot control method may include identifying whether the external object is the target object by inputting a sum of pixel values ​​of the region of interest into a first Gaussian probability distribution and a second Gaussian probability distribution. The robot control method may include identifying whether the external object is the target object based on a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution. The robot control method may include identifying the external object as the target object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is greater than or equal to a first critical value. The robot control method may include suspending judgment on the external object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a first critical value and equal to or greater than a second critical value smaller than the first critical value. The robot control method may include identifying the external object as not being the target object if a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a second critical value that is smaller than a first critical value. The robot control method may include training a plurality of classifiers included in the classifier group using a first training feature map for a tracked target and a second training feature map for a generic object. The robot control method may include obtaining the feature map by propagating a portion of the image through a plurality of convolution layers included in the neural network model. [Effects of the Invention]

[0011] According to the present invention, a camera and a lidar can be used to identify a target object, and identifying the target object in real time can be useful for path planning of a robot.

[0012] The present invention also allows for relatively inexpensive maintenance and repair of the robot by training a classifier using data acquired by the camera and lidar.

[0013] In addition, various other effects are provided that can be grasped directly or indirectly through this specification. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example of a robot control device according to an embodiment of the present invention. [Figure 2] FIG. 10 illustrates an example of obtaining a feature map in one embodiment of the present invention. [Figure 3] FIG. 2 illustrates an example of a classifier according to an embodiment of the present invention. [Figure 4] FIG. 1 illustrates an example of training a classifier in one embodiment of the present invention. [Figure 5] 10 is a diagram illustrating an example of determining whether an external object is a target object according to an embodiment of the present invention. [Figure 6] 1 is a flowchart illustrating an example of a robot control method according to an embodiment of the present invention. [Figure 7] 1 is a flowchart illustrating an example of a robot control method according to an embodiment of the present invention. [Figure 8] FIG. 1 illustrates a computing system for a robot control device or method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, specific examples of embodiments of the present invention will be described in detail with reference to the drawings.

[0016] When assigning reference numerals to components in each drawing, care should be taken to assign the same numerals to the same components even if they are displayed in different drawings. Furthermore, when describing embodiments of the present invention, if a detailed description of related known structures or functions is deemed to obscure understanding of the embodiments of the present invention, the detailed description will be omitted.

[0017] When describing components of embodiments of the present invention, terms such as "first," "second," "A," "B," "(a)," and "(b)" are used. These terms are used to distinguish a component from other components and do not limit the nature, order, or sequence of the components. Furthermore, unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. Terms similar to those defined in commonly used dictionaries should be interpreted as meanings consistent with the meanings they have in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0018] Hereinafter, an embodiment of the present invention will be described in detail with reference to FIGS.

[0019] FIG. 1 is a block diagram showing an example of a robot control device according to an embodiment of the present invention.

[0020] 1, a robot controller 100 according to an embodiment of the present invention may be implemented inside or outside the robot, and some of the components included in the robot controller 100 may be implemented inside or outside the robot. In this case, the robot controller 100 may be formed integrally with an internal control unit of the robot, or may be implemented as a separate device and connected to the control unit of the robot by a separate connection means. For example, the robot controller 100 may further include components not shown in FIG. 1.

[0021] The robot control device 100 according to this embodiment includes a processor 110, a lidar 120, a camera 130, and a memory 140. The processor 110, the lidar 120, the camera 130, and the memory 140 are electrically and / or operably coupled with each other by electronic components including a communication bus.

[0022] Hereinafter, when hardware is operatively coupled, it includes hardware in which a direct connection and / or an indirect connection between the hardware is established by wire and / or wirelessly, such that a first piece of hardware controls a second piece of hardware.

[0023] Although shown as different blocks, this embodiment is not limited thereto. Some of the hardware in FIG. 1 may be included in a single integrated circuit, including a system on a chip (SoC). The types and / or number of hardware included in the robot controller 100 are not limited to those shown in FIG. 1. For example, the robot controller 100 includes only a portion of the hardware shown in FIG. 1.

[0024] According to one embodiment, the robot controller 100 includes hardware for processing data based on one or more instructions. The hardware for processing data includes a processor 110. For example, the hardware for processing data includes an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor 110 may have a single-core processor structure or a multi-core processor structure, including a dual-core, quad-core, hexa-core, or octa-core structure.

[0025] The robot control device 100 according to this embodiment includes hardware for determining the distance between an external object and the robot. For example, the hardware for determining the distance between the external object and the robot includes a depth sensor. For example, the hardware for determining the distance between the external object and the robot includes at least one of a LIDAR 120, a time of flight (ToF) sensor, a structured light sensor, an ultrasonic sensor, an infrared sensor, an optical distance sensor, or any combination thereof.

[0026] For example, the LIDAR 120 of the robot controller 100 acquires a data set that identifies surrounding objects of the robot controller 100 (or a robot including the robot controller 100). For example, the LIDAR 120 identifies at least one of the position, direction of movement, speed, or any combination thereof of the surrounding objects based on a pulsed laser signal emitted from the LIDAR 120 being reflected back from the surrounding objects.

[0027] For example, the robot controller 100 acquires a data set representing an external object in a space defined by x-, y-, and z-axes based on a pulsed laser signal reflected from a surrounding object via the LIDAR 120. For example, the robot controller 100 acquires a data set including a plurality of points in a space defined by the x-, y-, and z-axes by receiving a pulsed laser signal at a specified period via the LIDAR 120.

[0028] The robot controller 100 according to this embodiment includes a camera 130. For example, the camera 130 includes one or more optical sensors (e.g., a charged couple device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals indicating the hue and / or brightness of light. The multiple optical sensors included in the camera 130 are arranged in the form of a two-dimensional array. The camera 130 acquires the electrical signals of each of the multiple optical sensors substantially simultaneously and generates an image or frame including multiple pixels arranged two-dimensionally in response to the light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera 130 refers to multiple images acquired from the camera 130. For example, video data captured using the camera 130 refers to a sequence of multiple images acquired from the camera 130 at a specified frame rate.

[0029] According to one embodiment, the memory 140 of the robot controller 100 includes hardware for storing data and / or instructions input to and / or output from the processor 110 of the robot controller 100 .

[0030] For example, memory 140 may include volatile memory, including random-access memory (RAM), and / or non-volatile memory, including read-only memory (ROM).

[0031] For example, the volatile memory includes at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, a pseudo SRAM (PSRAM), or any combination thereof.

[0032] For example, the non-volatile memory may include at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disk, a solid state drive (SSD), an embedded multi-media card (eMMC), or any combination thereof.

[0033] For example, the memory 140 may include a classifier group including a plurality of classifiers. For example, the memory 140 may include a neural network model. For example, the memory 140 may store a classifier group including a plurality of classifiers and / or a neural network model.

[0034] In this embodiment, the processor 110 acquires a point cloud corresponding to an external object via the lidar 120. For example, when the processor 110 acquires a point cloud corresponding to an external object via the lidar 120, the processor 110 projects the point cloud onto a specified plane. For example, the specified plane includes an xy plane in a three-dimensional spatial coordinate system formed by an x-axis, a y-axis, and a z-axis. For example, the x-axis points forward of the robot. For example, the y-axis points to the left side of the robot. For example, the z-axis is perpendicular to the ground.

[0035] For example, when the processor 110 acquires a point cloud corresponding to an external object via the lidar 120, the processor 110 acquires a virtual object represented in two dimensions by projecting the point cloud onto a specified surface.

[0036] For example, a virtual object represented in two dimensions corresponds to an external object, and is obtained by performing calibration with respect to a point cloud.

[0037] In this embodiment, the processor 110 acquires an image via the camera 130. For example, the processor 110 identifies a visual object that corresponds to a virtual object within the image acquired via the camera 130. For example, the visual object substantially corresponds to an external object.

[0038] For example, the processor 110 may identify a visual object corresponding to the virtual object in an image captured via the camera 130, and then input a portion of the image containing the visual object into a neural network model. For example, the neural network model may include multiple convolution layers. For example, the neural network model may include a model for acquiring at least one feature map for the image.

[0039] For example, the neural network model uses a convolution channel feature (CCF) method to output at least one feature map for the image.

[0040] For example, the processor 110 obtains the feature map by propagating a portion of the image through multiple convolution layers included in a neural network model.

[0041] In one embodiment, the processor 110 obtains a specified number of feature maps for a portion of an image from a neural network model. For example, the processor 110 obtains the feature maps output from the neural network model. For example, the processor 110 obtains the specified number of feature maps for a portion of an image from the neural network model, and then inputs the feature maps to a classifier group.

[0042] For example, the classifier group includes a plurality of classifiers. For example, each of the plurality of classifiers includes a first Gaussian probability distribution and a second Gaussian probability distribution. For example, the processor 110 trains the plurality of classifiers using a first training feature map for the tracked target and / or a second training feature map for the generic object. For example, the processor 110 trains the first Gaussian probability distribution using the first training feature map for the tracked target. For example, the processor 110 trains the second Gaussian probability distribution using the second training feature map for the generic object.

[0043] In one embodiment, the processor 110 acquires a specified number of feature maps for a portion of an image from a neural network model, and then inputs the acquired feature maps into a classifier group to identify whether an external object corresponding to a visual object is a target object.

[0044] For example, the target object is selected by the user through a screen displayed on a display (not shown) included in the robot control device 100. For example, when the robot control device 100 identifies a designated gesture of an external object, it sets the external object that has performed the designated gesture as the target object.

[0045] For example, the processor 110 identifies a region of interest in each of the feature maps input to the classifier group. For example, the processor 110 inputs each region of interest in each of the feature maps input to the classifier group to each of a plurality of classifiers. For example, the processor 110 identifies a region of interest in each of the feature maps input to the classifier group, and then inputs the region of interest to the plurality of classifiers to identify whether the external object is a target object.

[0046] For example, the processor 110 identifies pixel values ​​of the region of interest using each of the plurality of classifiers, and identifies whether the external object is a target object by identifying pixel values ​​of the region of interest using each of the plurality of classifiers.

[0047] For example, the processor 110 may identify pixel values ​​for the region of interest, such as a sum of pixel values ​​included in the region of interest, a sum of pixel values ​​with weights applied to the pixel values ​​included in the region of interest, an integration result for the pixel values ​​included in the region of interest, or any combination thereof.

[0048] For example, the processor 110 obtains a first result value output from a first Gaussian probability distribution, and for example, the processor 110 obtains a second result value output from a second Gaussian probability distribution.

[0049] For example, the processor 110 identifies a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution. For example, the processor 110 identifies the external object as a target object if the difference between the first result value output from the first Gaussian probability distribution and the second result value output from the second Gaussian probability distribution is equal to or greater than a first critical value.

[0050] For example, the first critical value may include a first percentage (e.g., about 70%). For example, the first critical value may include a first number. However, embodiments of the present invention are not limited to the above.

[0051] For example, the processor 110 may suspend judgment of the external object if the difference between the first result value output from the first Gaussian probability distribution and the second result value output from the second Gaussian probability distribution is less than a first critical value and greater than or equal to a second critical value that is smaller than the first critical value. For example, the second critical value may include a second percentage (e.g., about 40%). For example, the second critical value may include a second number that is less than the first number. However, embodiments of the present invention are not limited to the above.

[0052] For example, withholding judgment on an external object may involve a preceding process for outputting information about a target object identified at a previous point in time, when it is unclear whether the external object is a target object at the current point in time.

[0053] For example, the processor 110 identifies that the external object is not the target object if the difference between the first result value output from the first Gaussian probability distribution and the second result value output from the second Gaussian probability distribution is less than a second critical value.

[0054] The robot controller 100 according to this embodiment performs different operations depending on whether the external object is a target object. For example, if the external object is a target object, the robot controller 100 plans a path for the robot to follow the external object. For example, if the external object is not the target object, the robot controller 100 notifies the user that it has failed to follow the target object. For example, if the external object is not the target object, the robot controller 100 provides the user with at least one of a notification sound, a message, a screen display, or a combination thereof, indicating that it has failed to follow the target object.

[0055] As described above, the robot controller 100 according to the present embodiment can generate (or plan) a path for moving the robot controller 100 (or a robot including the robot controller 100) by identifying whether an external object is a target object using the LIDAR 120 and the camera 130. The processor of the robot controller 100 controls the robot using the generated path.

[0056] FIG. 2 is a diagram illustrating an example of obtaining a feature map in an embodiment of the present invention.

[0057] 2, a processor (e.g., processor 110 of FIG. 1) of a robot control device (e.g., robot control device 100 of FIG. 1) according to this embodiment acquires 3D object information 201 related to an external object via a rider (e.g., rider 120 of FIG. 1). For example, the processor acquires 3D object information 201 corresponding to the external object.

[0058] In this embodiment, the processor acquires an image 202 including a visual object corresponding to an external object via a camera (e.g., camera 130 in FIG. 1). For example, the processor projects the 3D object information 201 onto a specified plane (e.g., the yz plane). For example, the processor identifies a virtual object generated from the 3D object information 201 by projecting the 3D object information 201 onto the specified plane. For example, the processor identifies a visual object in the image 202 that corresponds to the virtual object generated from the 3D object information 201.

[0059] In this embodiment, the processor identifies the portion of the image 203 that includes the visual object. For example, the processor identifies the portion of the image 203 that includes the visual object by identifying a visual object in the image 202 that corresponds to a virtual object generated by the three-dimensional object information 201.

[0060] For example, after identifying image portion 203, processor segments image portion 203. For example, processor segments image portion 203 in order to input image portion 203 into neural network model 210 stored in a memory (e.g., memory 140 in FIG. 1 ).

[0061] In this embodiment, the processor inputs the portion of the image 203 to the neural network model 210. For example, the processor inputs the portion of the image 203 to the neural network model 210 to obtain a feature map 230 for the portion of the image 203.

[0062] For example, the neural network model 210 includes multiple convolution layers (211, 212). For example, the first layer 211 of the multiple convolution layers (211, 212) included in the neural network model 210 has a size of 5×5×3 and includes 20 channels. For example, the second layer 212 of the multiple convolution layers (211, 212) included in the neural network model 210 has a size of 5×5×20 and includes 25 channels.

[0063] In this embodiment, the processor obtains a specified number of feature maps 230 by inputting a portion of an image 203 into a neural network model 210 including multiple convolutional layers (211, 212). For example, the specified number corresponds to the number of channels in the second convolutional layer 212. For example, the specified number includes approximately 25.

[0064] As described above, the processor of the robot controller according to this embodiment can use a neural network model to obtain a feature map 230 for visual objects contained in the portion of the image 203 .

[0065] FIG. 3 is a diagram illustrating an example of a classifier according to an embodiment of the present invention.

[0066] Referring to FIG. 3, a processor (e.g., processor 110 in FIG. 1) of a robot control device (e.g., robot control device 100 in FIG. 1) according to this embodiment trains a classifier group set 310 including a classifier group 311 stored in a memory (e.g., memory 140 in FIG. 1), or identifies (or tracks) a target object using the classifier group set 310.

[0067] For example, the classifier group set 310 includes multiple classifier groups 311. For example, the multiple classifier groups 311 include multiple classifiers. For example, the classifier group set 310 includes a machine learning algorithm. For example, the classifier group set 310 uses online boosting.

[0068] For example, the classifier group set 310 includes n classifier groups 311. For example, each of the n classifier groups 311 includes m classifiers 320.

[0069] For example, the classifier 320 includes a first Gaussian probability distribution 321 and a second Gaussian probability distribution 322. For example, the classifier 320 determines whether an external object is a target object using the first Gaussian probability distribution 321 and the second Gaussian probability distribution 322.

[0070] For example, the first Gaussian probability distribution 321 includes a probability distribution for determining whether an external object corresponds to a target object, and the second Gaussian probability distribution 322 includes a probability distribution for determining whether an external object does not correspond to a target object.

[0071] For example, the processor inputs the feature map obtained from the neural network model into the classifier group set 310 to identify whether the external object corresponds to the target object using the first Gaussian probability distribution 321 and the second Gaussian probability distribution 322.

[0072] As described above, the processor of the robot control device according to the present embodiment can control a robot including the robot control device by identifying whether an external object corresponds to a target object.

[0073] FIG. 4 is a diagram illustrating an example of training a classifier in one embodiment of the present invention.

[0074] 4, a processor (e.g., processor 110 in FIG. 1) of a robot control device according to this embodiment (e.g., robot control device 100 in FIG. 1) trains a classifier 410. For example, the processor trains a first Gaussian function distribution 411 and / or a second Gaussian function distribution 412 included in the classifier 410.

[0075] For example, the classifier 410 is included in the classifier group described in Fig. 3. For example, the classifier 410 includes a first classifier 410-1, a second classifier 410-2, ..., and M classifiers 410-n.

[0076] For example, the processor trains a classifier 410 using the first training feature map 401 and the second training feature map 402. For example, a first classifier 410-1 of the classifiers 410 includes a first Gaussian probability distribution 411-1 and a second Gaussian probability distribution 412-1. For example, a second classifier 410-2 of the classifiers 410 includes a first Gaussian probability distribution 411-2 and a second Gaussian probability distribution 412-2. For example, an Mth classifier 410-n of the classifiers 410 includes a first Gaussian probability distribution 411-n and a second Gaussian probability distribution 412-n.

[0077] For example, each of the classifiers 410 determines whether a region of interest included in a portion of an image matches a target object. For example, the processor uses the classifiers 410 to determine whether a region of interest matches a target object.

[0078] For example, the first Gaussian probability distribution 411 and the second Gaussian probability distribution 412 included in each of the classifiers 410 each include a probability distribution for determining whether a designated region of interest matches a target object.

[0079] For example, if the input data (e.g., a training feature map) corresponds to a target object, the processor updates the first Gaussian probability distribution 411. For example, if the input data does not correspond to a target object, the processor updates the second Gaussian probability distribution 412. For example, the first Gaussian probability distribution 411 is referred to as a positive probability distribution. For example, the second Gaussian probability distribution 412 is referred to as a negative probability distribution.

[0080] In this embodiment, the processor trains the classifier 410 and selects the best-trained classifier as the representative classifier. For example, the best-trained classifier relates to whether the first Gaussian probability distribution and the second Gaussian probability distribution are clearly separated. For example, the best-trained classifier includes a classifier in which the first Gaussian probability distribution and the second Gaussian probability distribution are clearly separated.

[0081] As described above, the processor of the robot control device according to this embodiment can accurately determine whether an external object corresponding to a visual object included in a part of an image is a target object by training the classifier 410.

[0082] FIG. 4 illustrates an example of training the classifier 410. Using the classifier 410 to determine whether an external object corresponds to a target object is essentially the same as that described above.

[0083] For example, the processor uses a first classifier 410-1 to determine whether a first region of interest matches a target object, and uses a second classifier 410-2 to determine whether a second region of interest matches a target object.

[0084] The processor also identifies whether an external object corresponding to a visual object included in a portion of the image is a target object based on the output of the classifier 410. For example, if the processor determines that approximately 70% or more of the outputs (i.e., result values) of the classifier 410 indicate that the external object is a target object, the processor determines the external object as the target object. For example, if the processor determines that approximately 40% to less than 70% of the outputs of the classifier 410 indicate that the external object is a target object, the processor suspends its determination of the external object. For example, if the processor determines that approximately 40% or less than 40% of the outputs of the classifier 410 indicate that the external object is a target object, the processor determines that the external object is not a target object.

[0085] FIG. 5 is a diagram illustrating an example of determining whether an external object is a target object according to an embodiment of the present invention.

[0086] Referring to FIG. 5, a processor (eg, processor 110 in FIG. 1) of a robot control device (eg, robot control device 100 in FIG. 1) according to this embodiment performs filtering using a standard deviation 510.

[0087] A first example 501 of FIG. 5 includes examples of a first Gaussian probability distribution and a second Gaussian probability distribution. In the first example 501, the value represented as 24 represents the sum of pixel values. In the first example 501, it is confirmed that the sum of pixel values ​​is included in the first Gaussian probability distribution from the first Gaussian probability distribution and the second Gaussian probability distribution. For example, if the sum of pixel values ​​is included in the first Gaussian probability distribution from the first Gaussian probability distribution and the second Gaussian probability distribution, it indicates that an external object corresponding to a visual object included in a portion of an image is identified as a target object.

[0088] In the second example 502 of FIG. 5, the value represented as 70 indicates the sum of pixel values. In the second example 502, it is confirmed that the sum of pixel values ​​is included in the second Gaussian probability distribution from the first Gaussian probability distribution and the second Gaussian probability distribution. For example, if the sum of pixel values ​​is included in the second Gaussian probability distribution from the first Gaussian probability distribution and the second Gaussian probability distribution, it is identified that an external object corresponding to a visual object included in a portion of the image is not the target object.

[0089] In this embodiment, the processor determines whether the external object corresponds to the target object using the sum and standard deviation of pixel values ​​510. For example, if the sum of pixel values ​​falls outside a specified range of the standard deviation 510, the processor does not use the sum of pixel values ​​to determine whether the external object corresponds to the target object.

[0090] FIG. 6 is a flowchart showing an example of a robot control method according to an embodiment of the present invention.

[0091] Hereinafter, it is assumed that the robot controller 100 of Fig. 1 performs the process of Fig. 6. Furthermore, in the explanation of Fig. 6, the operations described as being performed by the device are understood to be controlled by the processor 110 of the robot controller 100.

[0092] At least one of the steps in Fig. 6 is performed by the robot control device 100 of Fig. 1. At least one of the steps in Fig. 6 is controlled by the processor 110 of Fig. 1. The steps in Fig. 6 are performed sequentially, but not necessarily sequentially. For example, the order of the steps may be changed, or at least two steps may be performed in parallel.

[0093] Referring to FIG. 6, the robot control method according to this embodiment includes an operation of acquiring data on an external object via a lidar 601 and / or a camera 602.

[0094] In step S601, the robot control method according to this embodiment includes an operation of acquiring a point cloud via a lidar 601. For example, the point cloud includes a set of points corresponding to an external object in a three-dimensional virtual coordinate system.

[0095] In step S602, the robot control method according to this embodiment includes acquiring geometric information of the object. For example, the robot control method may include acquiring geometric information of the object using a point cloud. For example, the geometric information may include at least one of the object's speed, the object's heading direction, the object's size, or any combination thereof.

[0096] In step S603, the robot control method according to this embodiment includes an operation of projecting LIDAR points. For example, the LIDAR points may be included in a point cloud. For example, the LIDAR points may include points corresponding to external objects acquired by the LIDAR 601.

[0097] In step S604, the robot control method according to this embodiment includes an operation of extracting a candidate object. For example, the robot control method includes an operation of extracting a candidate object within an image acquired via the camera 602. For example, the candidate object may include an external object.

[0098] In step S605, the robot control method according to this embodiment includes extracting a feature map, for example, by inputting at least a portion of an image including a candidate object into a neural network model.

[0099] In step S606, the robot control method according to this embodiment includes an operation of inputting the feature map to the classifier group. For example, the robot control method includes an operation of inputting the feature map output from the neural network model to the classifier group.

[0100] In step S607, the robot control method according to this embodiment includes an operation of outputting a probability and geometric information of the object. For example, the robot control method includes an operation of outputting a probability that a visual object corresponding to the external object is a target object and geometric information about the external object based on the feature map.

[0101] In step S608, the robot control method according to this embodiment includes determining whether there are one or more targets with a probability exceeding 70%. For example, the robot control method includes determining whether there are one or more targets with a probability exceeding 70% that a visual object corresponding to an external object is the target object.

[0102] If there is one or more targets with a probability exceeding 70% (Yes in step S608), the robot control method according to this embodiment includes, in step S609, an operation of selecting an object closest to the position of the previous target as the target.

[0103] In step S610, the robot control method according to this embodiment includes an operation of training a classifier. For example, the robot control method may include an operation of selecting an object closest to the position of a previous target as a target and training a classifier using the selected target.

[0104] If there are not at least one target with a probability exceeding 70% (No in step S608), the process proceeds to step S612, where it is determined whether an obstacle exists around the target for the last 0.5 seconds. Although the last 0.5 seconds has been described, the present embodiment is not limited to this.

[0105] For example, since the robot control method performs an operation of about 10 frames (ie, 10 times) per second, the most recent 0.5 seconds means about 5 frames.

[0106] If no obstacle has been present around the target for the last 0.5 seconds (No in step S612), the robot control method according to this embodiment includes an operation of training the classifier in step S610.

[0107] In step S611, the robot control method according to this embodiment includes an operation of outputting position information of the target object, for example, in a frame acquired at a current time point.

[0108] If an obstacle has been present around the target for the past 0.5 seconds (Yes in step S612), the robot control method according to this embodiment includes an operation of suspending judgment and saving data at the current time in step S613. For example, the robot control method does not determine whether a visual object included in the image of the currently acquired frame corresponds to the target object.

[0109] In step S614, the robot control method according to the present embodiment includes an operation of counting the number of times a decision has been suspended, for example, the robot control method includes an operation of counting the number of times a decision has been suspended at the current time point.

[0110] In step S615, the robot control method according to this embodiment includes an operation of determining whether the count exceeds 100. For example, the robot control method includes an operation of counting the number of times the determination is suspended and determining whether the count exceeds 100.

[0111] For example, a count exceeding 100 times includes 100 frames in which the target object is not identified.

[0112] If the count exceeds 100 (Yes in step S615), the robot control method according to this embodiment includes an operation of declaring a tracking failure in step S620. For example, the robot control method may include an operation of declaring a tracking failure due to a failure to track the target object. For example, a failure to track the target object may include an inability to identify the target object in the image.

[0113] If the count does not exceed 100 (No in step S615), the robot control method according to this embodiment includes an operation of outputting past position information of the target object in step S616.

[0114] In step S616, past means the frame immediately before which the decision is deferred.

[0115] In step S617, the robot control method according to this embodiment includes an operation of acquiring terrain information. For example, the robot control method includes an operation of acquiring terrain information based on a point cloud and / or points acquired by the LIDAR 601. For example, the terrain information includes a map representing the surrounding environment of the robot controller (or a robot including the robot controller).

[0116] In step S618, the robot control method according to this embodiment includes an operation of planning a path for the robot to travel, for example, based on terrain information and / or position information of a target object.

[0117] In step S619, the robot control method according to this embodiment includes an operation of generating a robot control signal. For example, the robot control method includes an operation of generating a signal for moving the robot according to the planned path. For example, the robot control method includes an operation of moving (or controlling) the robot based on the generated signal.

[0118] FIG. 7 is a flowchart showing an example of a robot control method according to an embodiment of the present invention.

[0119] Hereinafter, it is assumed that the robot controller 100 of Fig. 1 performs the process of Fig. 7. Furthermore, in the explanation of Fig. 7, the operations described as being performed by the device are understood to be controlled by the processor 110 of the robot controller 100.

[0120] At least one of the steps in Fig. 7 is performed by the robot controller 100 of Fig. 1. At least one of the steps in Fig. 7 is controlled by the processor 110 of Fig. 1. The steps in Fig. 7 are performed sequentially, but not necessarily sequentially. For example, the order of the steps may be changed, or at least two steps may be performed in parallel.

[0121] In step S701, the robot control method according to this embodiment includes an operation of acquiring a point cloud corresponding to an external object through a lidar, and then projecting the point cloud onto a specified surface to acquire a virtual object represented in two dimensions.

[0122] In step S703, the robot control method according to this embodiment includes identifying a visual object corresponding to a virtual object in an image acquired via a camera, and then inputting a portion of the image including the visual object into a neural network model.

[0123] For example, a robot control method may include obtaining a feature map by propagating a portion of an image through multiple convolution layers included in a neural network model.

[0124] In step S705, the robot control method according to this embodiment includes an operation of acquiring a specified number of feature maps for a portion of an image from a neural network model, and then inputting the feature maps into a classifier group to identify whether an external object corresponding to a visual object is a target object.

[0125] For example, the robot control method may include identifying whether an external object is a target object based on each region of interest of the feature map input to the classifier group.

[0126] For example, the robot control method may include an operation of identifying pixel values ​​of a region of interest using each of a plurality of classifiers included in a classifier group, and an operation of identifying whether an external object is a target object by identifying pixel values ​​of the region of interest using each of the plurality of classifiers.

[0127] For example, the robot control method may include identifying whether the external object is a target object by inputting a sum of pixel values ​​of a region of interest into a first Gaussian probability distribution and a second Gaussian probability distribution.

[0128] For example, the robot control method includes an operation of identifying whether an external object is a target object based on a first result value output from a first Gaussian probability distribution and a second result value output from a second Gaussian probability distribution. For example, the robot control method includes an operation of identifying whether an external object is a target object based on a first result value obtained from the first Gaussian probability distribution and a second result value obtained from the second Gaussian probability distribution.

[0129] For example, the robot control method may include an operation of comparing a difference between a first result value and a second result value, and an operation of identifying whether the external object is a target object by comparing the difference between the first result value and the second result value.

[0130] For example, the robot control method may include identifying the external object as the target object if a difference between the first result value and the second result value is equal to or greater than a first critical value.

[0131] For example, the robot control method includes suspending judgment on the external object when the difference between the first result value and the second result value is less than a first critical value and equal to or greater than a second critical value that is smaller than the first critical value.

[0132] For example, the robot control method may include identifying the external object as not being the target object if the difference between the first result value and the second result value is less than a second critical value.

[0133] A robot control method according to this embodiment includes an operation of training a plurality of classifiers using a first training feature map for the tracked target and / or a second training feature map for the generic object. For example, the robot control method includes an operation of training a classifier for a first Gaussian probability distribution using the first training feature map for the tracked target. For example, the robot control method includes an operation of training a classifier for a second Gaussian probability distribution using the second training feature map for the generic object.

[0134] FIG. 8 is a diagram illustrating a computing system relating to a robot control device or a robot control method according to an embodiment of the present invention.

[0135] Referring to FIG. 8, computing system 1000 includes at least one processor 1100, memory 1300, user interface input device 1400, user interface output device 1500, storage 1600, and network interface 1700, all connected via a bus 1200.

[0136] The processor 1100 is a semiconductor device that executes processing based on instructions stored in a central processing unit (CPU), memory 1300, and / or storage 1600. The memory 1300 and storage 1600 may include various types of volatile or non-volatile recording media. For example, the memory 1300 may include a read only memory (ROM) 1310 and a random access memory (RAM) 1320.

[0137] Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware executed by processor 1100, in a software module, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or other recording medium (i.e., memory 1300 and / or storage 1600).

[0138] An exemplary storage medium is coupled to processor 1100, such that processor 1100 reads information from and records information on the storage medium. Alternatively, the storage medium may be integral to processor 1100. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

[0139] The above description is merely an illustrative example of the technical concept of the present invention, and various modifications and variations can be made by a person having ordinary knowledge in the technical field to which the present invention pertains without departing from the essential characteristics of the present invention.

[0140] Therefore, the embodiments disclosed in the present invention are for illustrative purposes only and do not limit the technical idea of ​​the present invention, and the scope of the technical idea of ​​the present invention should not be limited by such embodiments. The scope of protection of the present invention should be interpreted by the claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention. [Explanation of symbols]

[0141] 100 Robot control device 110, 1100 processors 120, 601 riders 130, 602 Camera 140, 1300 memory 201 3D object information 202 images 203 Part of the image 210 Neural Network Model 211 Convolution Layer (1st Layer) 212 Convolution Layer (Second Layer) 230 feature maps 310 Classifier Group Set 311 Classifier Group 320, 410 classifier 321, 322 First and second Gaussian probability distributions 401, 402 First and second learning feature maps 410-1~410-n 1st~M classifier 411, 412 First and second Gaussian function distribution 411-1~411-n 1st to Mth Gaussian probability distributions 412-1~412-n 1st to Mth Gaussian probability distributions 501, 502 First and second examples 510 standard deviations 1000 Computing Systems 1200 Bus 1310 ROM 1320 RAM 1400 User Interface Input Device 1500 User interface output device 1600 Storage 1700 network interface

Claims

1. LiDAR (light detection and ranging); A camera and a memory in which a classifier group including a plurality of classifiers and a neural network model are stored; a processor, The processor: When a point cloud corresponding to an external object is acquired through the lidar, the point cloud is projected onto a designated surface to acquire a virtual object represented in two dimensions; identifying a visual object corresponding to the virtual object in an image acquired through the camera, and then inputting a portion of the image including the visual object into the neural network model; and acquiring a designated number of feature maps for a portion of the image from the neural network model, and inputting the feature maps to the classifier group to identify whether the external object corresponding to the visual object is a target object.

2. 2. The robot control device of claim 1, wherein the processor is configured to identify whether the external object is the target object based on a region of interest (ROI) of each of the feature maps input to the classifier group.

3. The robot control device of claim 2 , wherein the processor is configured to identify whether the external object is the target object by identifying pixel values ​​of the region of interest using each of the plurality of classifiers.

4. 4. The robot control device of claim 3, wherein the processor is configured to identify whether the external object is the target object by inputting a sum of pixel values ​​of the region of interest into a first Gaussian probability distribution and a second Gaussian probability distribution.

5. 5. The robot control device of claim 4, wherein the processor is configured to identify whether the external object is the target object based on a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution.

6. 5. The robot control device of claim 4, wherein the processor is configured to identify the external object as the target object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is equal to or greater than a first critical value.

7. 5. The robot control device of claim 4, wherein the processor is configured to suspend judgment on the external object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a first critical value and equal to or greater than a second critical value smaller than the first critical value.

8. 5. The robot control device of claim 4, wherein the processor is configured to identify that the external object is not the target object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a second critical value that is smaller than a first critical value.

9. The robot controller of claim 1 , wherein the processor is configured to train the plurality of classifiers using a first training feature map for a tracked target and a second training feature map for a generic object.

10. The robot controller of claim 1 , wherein the processor is configured to obtain the feature map by propagating a portion of the image through a plurality of convolution layers included in the neural network model.

11. 2. The robot control device according to claim 1, wherein the processor is configured to initialize at least one of the partial classifiers excluding the representative classifier when the representative classifier is selected from the plurality of classifiers.

12. The robot controller of claim 1 , wherein the processor is configured to assign a region of interest based on at least one of the feature maps while initializing at least one of the plurality of classifiers.

13. 1. A processor-implemented robot control method, comprising: When the processor acquires a point cloud corresponding to an external object through light detection and ranging (LiDAR), the processor projects the point cloud onto a designated surface to acquire a virtual object represented in two dimensions; identifying a visual object corresponding to the virtual object in an image acquired via a camera, and then inputting a portion of the image including the visual object into a neural network model stored in a memory; acquiring a designated number of feature maps for a portion of the image from the neural network model, and then inputting the feature maps into a classifier group stored in the memory to identify whether the external object corresponding to the visual object is a target object.

14. 14. The method of claim 13, further comprising identifying whether the external object is the target object based on each region of interest (ROI) of the feature map input to the classifier group.

15. 15. The robot control method of claim 14, further comprising identifying whether the external object is the target object by identifying pixel values ​​of the region of interest using each of a plurality of classifiers included in the classifier group.

16. 16. The robot control method of claim 15, further comprising inputting a sum of pixel values ​​of the region of interest into a first Gaussian probability distribution and a second Gaussian probability distribution to identify whether the external object is the target object.

17. 17. The method of claim 16, further comprising: identifying whether the external object is the target object based on a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution.

18. 17. The method of claim 16, further comprising identifying the external object as the target object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is equal to or greater than a first critical value.

19. 17. The robot control method of claim 16, further comprising suspending judgment on the external object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a first critical value and equal to or greater than a second critical value smaller than the first critical value.

20. 17. The robot control method of claim 16, further comprising identifying the external object as not being the target object when a difference between a first result value output from the first Gaussian probability distribution and a second result value output from the second Gaussian probability distribution is less than a second critical value that is smaller than a first critical value.

21. 14. The robot control method of claim 13, further comprising training a plurality of classifiers included in the classifier group using a first training feature map for a tracked target and a second training feature map for a generic object.

22. 14. The robot control method of claim 13, further comprising: acquiring the feature map by propagating a portion of the image through a plurality of convolution layers included in the neural network model.

Images