Robot and control method thereof

Abstract

A robot includes: a light detection and ranging (LiDAR) sensor; a plurality of directional microphones; and at least one processor configured to identify an object in the vicinity of the robot based on sensing data obtained by the LiDAR sensor, identify a weight applied to an audio signal received through a directional microphone corresponding to a position of the object among the plurality of directional microphones based on a type of the object, obtaining contextual information of the robot based on the sensed data, identifying a preprocessing model corresponding to each of the plurality of directional microphones based on the contextual information, applying a weight to an audio signal received by a directional microphone corresponding to the position of the object among a plurality of audio signals received by the plurality of directional microphones, a plurality of preprocessed audio signals are obtained by inputting the audio signal to which the weight has been applied and the remaining audio signals into preprocessing models corresponding to the respective directional microphones, and speech recognition is performed based on the plurality of preprocessed audio signals.

Description

Technical Field

[0001] The present disclosure relates to a robot and a control method thereof, and more particularly, to a robot which can control a microphone included in the robot based on sensing data obtained using a Light Detection and Ranging (LiDAR) sensor and a control method thereof. Background Art

[0002] In recent years, with the development of electronic technology, robots have been widely used not only in specialized technical fields or industrial fields that require large-scale labor, but also in general homes and shops. For example, household cleaning robots, guidance robots that guide users in stores, service robots in stores, etc. can belong to this category.

[0003] According to the development of voice recognition technology, users can recently control robots through user voices without inputting control commands through a separate control device or interface. For voice recognition, the robot needs to receive clear user voices. However, it may happen that the robot fails to fully receive the user voice due to the surrounding environment of the robot. For example, a user at a specific location may speak when the robot is surrounded by walls, and the robot may receive the user's voice and noise reflected from the walls. In another example, the user's voice may be distorted due to noise occurring around the robot.

[0004] When performing a speech processing process (such as user position recognition or speech recognition) based on speech including noise or distorted speech, the robot may obtain incorrect results. That is, the robot may erroneously recognize the position of a user who attempts to verbally control the robot, or obtain a speech recognition result that is completely different from the user's intention. Therefore, there is a need for a method of obtaining an improved audio signal by processing an audio signal received by a robot in consideration of information about the robot's surroundings. Summary of the invention

[0005] [Technical solution] According to one aspect of the present disclosure, a robot includes a light detection and ranging (LiDAR) sensor; a plurality of directional microphones; and at least one processor, configured to: identify objects near the robot based on sensing data obtained by the LiDAR sensor, identify weights applied to audio signals received by a directional microphone among the plurality of directional microphones corresponding to a position of the object based on the type of the object, obtain context information of the robot based on the sensing data, identify a preprocessing model corresponding to each of the plurality of directional microphones based on the context information, apply the weights to audio signals received by the directional microphone corresponding to the position of the object among a plurality of audio signals received by the plurality of directional microphones, obtain a plurality of preprocessed audio signals by inputting the audio signals to which the weights have been applied and the remaining audio signals into the preprocessing models corresponding to the respective directional microphones, and perform speech recognition based on the plurality of preprocessed audio signals.

[0006] The at least one processor may also be configured to, based on identifying the object as a person, identify a first value as the weight applied to the audio signal received through the directional microphone corresponding to the position of the object, and based on identifying the object as a wall, identify a second value as the weight applied to the audio signal received through the directional microphone corresponding to the position of the object, and wherein the first value is greater than the second value.

[0007] The at least one processor may also be configured to obtain first contextual information about the space near the robot and second contextual information about the driving state of the robot based on sensing data, and to identify a preprocessing model corresponding to each of the multiple directional microphones based on at least one of the first contextual information and the second contextual information.

[0008] The first context information includes at least one of information about the type of the object, information about the location of the object, information about the number of objects near the robot, and information about the distance between the robot and the object, and wherein the second context information includes at least one of information about whether the robot is moving, information about the moving speed of the robot, and information about the moving direction of the robot.

[0009] The at least one processor may be further configured to: recognize that the robot is moving based on second contextual information, and based on recognizing that the robot is moving, identify the preprocessing model including the first model as a preprocessing model corresponding to each of the plurality of directional microphones.

[0010] The at least one processor may also be configured to: based on identifying that the robot is moving toward the object identified as a person according to the first contextual information and the second contextual information, identify the preprocessed model including the first model and the second model as a preprocessed model corresponding to the directional microphone corresponding to the position of the object, obtain a first preprocessed audio signal by inputting the audio signal to which the weight of the first value is applied into the first model, and obtain a second preprocessed audio signal by inputting the first preprocessed audio signal into the second model.

[0011] The at least one processor may be further configured to: based on identifying the object as a wall adjacent to the robot according to the first contextual information, identify a preprocessing model including a third model as a preprocessing model corresponding to each of the plurality of directional microphones.

[0012] The at least one processor may be further configured to identify a preprocessing model including a fourth model as a preprocessing model corresponding to each of the plurality of directional microphones based on identifying a plurality of objects near the robot according to the first contextual information and the plurality of objects being a plurality of persons.

[0013] According to one aspect of the present disclosure, a method for controlling a robot includes: identifying an object near the robot based on sensing data obtained by a light detection and ranging (LiDAR) sensor; identifying a weight to be applied to an audio signal received by a directional microphone corresponding to the position of the object among a plurality of directional microphones based on the type of the object; obtaining context information of the robot based on the sensing data; identifying a preprocessing model corresponding to each corresponding directional microphone among the plurality of directional microphones based on the context information; applying the weight to an audio signal received by the directional microphone corresponding to the position of the object among a plurality of audio signals received by the plurality of directional microphones; obtaining a plurality of preprocessed audio signals by inputting the audio signal to which the weight has been applied and the remaining audio signals into the preprocessing models corresponding to the respective directional microphones; and performing speech recognition based on the plurality of preprocessed audio signals.

[0014] Identifying the weight may also include: based on identifying the object as a person, identifying a first value as the weight applied to the audio signal received through the directional microphone corresponding to the position of the object; and based on identifying the object as a wall, identifying a second value as the weight applied to the audio signal received through the directional microphone corresponding to the position of the object, and wherein the first value is greater than the second value.

[0015] Obtaining contextual information may also include: obtaining first contextual information about the space near the robot and second contextual information about the driving state of the robot based on sensing data, and wherein the method further includes: identifying a preprocessing model corresponding to each of the multiple directional microphones based on at least one of the first contextual information and the second contextual information.

[0016] The first context information includes at least one of information about the type of the object, information about the location of the object, information about the number of objects near the robot, and information about the distance between the robot and the object, and wherein the second context information includes at least one of information about whether the robot is moving, information about the moving speed of the robot, and information about the moving direction of the robot.

[0017] Identifying the preprocessing model further includes: identifying that the robot is moving based on the second context information, and based on identifying that the robot is moving, identifying the preprocessing model including the first model as a preprocessing model corresponding to each of the plurality of directional microphones.

[0018] Identifying the preprocessing model may also include: based on identifying that the robot is moving toward the object identified as a person according to the first contextual information and the second contextual information, identifying the preprocessing model including the first model and the second model as a preprocessing model corresponding to the directional microphone corresponding to the position of the object, and wherein obtaining the multiple preprocessed audio signals also includes: obtaining a first preprocessed audio signal by inputting an audio signal to which the weight of the first value is applied into the first model; and obtaining a second preprocessed audio signal by inputting the first preprocessed audio signal into the second model.

[0019] Identifying the preprocessing model may further include identifying a preprocessing model including a third model as a preprocessing model corresponding to each of the plurality of directional microphones based on identifying the object as a wall adjacent to the robot according to the first context information.

[0020] According to an aspect of the present disclosure, a computer-readable recording medium recording a computer program for executing the method may also be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which: Figure 1 is an exemplary diagram for explaining the operation of the robot according to an embodiment of the present disclosure; Figure 2is a schematic configuration diagram of a robot according to an embodiment of the present disclosure; Figure 3 is a diagram for explaining a plurality of directional microphones according to an embodiment of the present disclosure; Figure 4 is an exemplary diagram showing information about the surrounding environment of a robot obtained through a LiDAR sensor according to an embodiment of the present disclosure; Figure 5 is an exemplary diagram showing a weight of identifying a directional microphone when an object around a robot is identified as a person according to an embodiment of the present disclosure; Figure 6 is an exemplary diagram showing a weight of identifying a directional microphone when an object around a robot is identified as a wall according to an embodiment of the present disclosure; Figure 7 is an exemplary diagram showing weights of identifying directional microphones when objects around a robot are identified as people and walls according to an embodiment of the present disclosure; Figure 8 is a diagram illustrating a method for robot-based context information recognition preprocessing model according to an embodiment of the present disclosure; Fig. 9 is an exemplary diagram showing a preprocessing model applied to a plurality of directional microphones when a robot is identified as traveling according to an embodiment of the present disclosure; Fig.10 is an exemplary diagram showing a preprocessing model applied to a plurality of directional microphones when a robot is identified as traveling toward a person according to an embodiment of the present disclosure; Fig.11 is an exemplary diagram illustrating a preprocessing model applied to a plurality of directional microphones when a robot is identified as traveling toward a person adjacent to a wall according to an embodiment of the present disclosure; Fig.12 is a detailed configuration diagram of a robot according to an embodiment of the present disclosure; and Fig.13 is a flowchart schematically illustrating a method for controlling a robot according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The terms used in the specification or claims are selected from general terms in consideration of their functions in various embodiments of the present disclosure. However, these terms may be changed based on the intentions of technicians in the field to which the present disclosure belongs, legal or technical interpretations, and the emergence of new technologies. In addition, some terms are arbitrarily selected by the applicant. The term may be interpreted as having the meaning defined in the specification, and if there is no specific definition of the term, the term may be interpreted based on the general content of the specification and the common technical knowledge in the art.

[0023] In addition, in the drawings throughout the specification, the same reference numerals represent parts or components that perform substantially the same function. For ease of description and understanding, different embodiments are described using the same reference numerals. That is, even if all components with the same reference numerals are shown in multiple drawings, the multiple drawings do not show one embodiment.

[0024] In addition, in the specification and claims, terms including ordinal numbers (such as "first", "second", etc.) may be used to distinguish various components from each other. Such ordinal numbers may be used only to distinguish the same or similar components from each other and should not be interpreted as being limited to these terms. For example, components combined with these ordinal numbers should not be limited to their order of use or placement. If desired, ordinal numbers can be used interchangeably.

[0025] In the specification, unless the context clearly indicates otherwise, the singular form of the term includes its plural form. It will be understood that the terms "comprising", "formed by...", etc. used in this application specify the presence of the features, numbers, steps, operations, components, parts or their combinations mentioned in the specification, and do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or their combinations.

[0026] In the embodiments of the present disclosure, terms such as "module", "unit", "component", etc. are used to refer to a component that performs at least one function or operation. The component may be implemented by hardware or software or by a combination of hardware and software. In addition, in addition to each of the multiple "modules", "units", "components", etc. that need to be implemented by specific hardware, the multiple "modules", "units", "components", etc. may be integrated in at least one module or chip and implemented by at least one processor.

[0027] In addition, in the embodiments of the present disclosure, it will be understood that when one component is referred to as being connected to another component, the one component may not only be directly connected to the other component but may also be indirectly connected to the other component through a third medium. In addition, unless otherwise explicitly described, any statement implying the inclusion of any component is understood to imply the inclusion of other components rather than excluding any other components.

[0028] As used herein, expressions such as "at least one of..." when preceding a list of elements modify the entire list of elements rather than modifying the individual elements of the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0029] Hereinafter, various embodiments of the present disclosure are described in detail with reference to the accompanying drawings.

[0030] Figure 1 2 is an exemplary diagram for explaining the operation of the robot according to an embodiment of the present disclosure.

[0031] Reference Figure 1 , the robot 100 according to an embodiment of the present disclosure may obtain information about an object around (near) the robot 100 through the LiDAR sensor 110. The information about the object may include the type of the object and information about the distance between the robot and the object.

[0032] Here, when objects located around the robot 100 are respectively recognized as the person 10 and the wall 20 , the robot 100 may predict that an audio signal, particularly an audio signal including a voice command for controlling the robot 100 , will be output from the object recognized as the person 10 .

[0033] Therefore, the robot 100 may increase the strength of an audio signal obtained by a directional microphone in a direction corresponding to the position of an object recognized as the person 10, among audio signals received by the plurality of directional microphones 120. This configuration is to perform voice recognition more smoothly by increasing the strength of an audio signal predicted to include a voice command for controlling the robot 100.

[0034] In addition, the robot 100 may recognize the surrounding environment of the robot 100 to recognize a preprocessing model for preprocessing the received audio signal. Figure 1 As shown, the robot 100 is adjacent to a wall. In this case, the robot 100 may receive not only a voice directly uttered by the user but also noise that occurs when the user voice is reflected from the wall. Therefore, the robot 100 may be required to perform voice recognition or voice processing for identifying the user's position through a distorted user voice including noise. This configuration may result in the robot 100 obtaining an incorrect voice recognition result or recognizing the user's position as an incorrect position.

[0035] Therefore, the robot 100 according to an embodiment of the present disclosure may identify an appropriate preprocessing model to be applied to an audio signal received by the robot 100 in consideration of the surrounding environment of the robot 100. Figures 2 to 13 Embodiments of the present disclosure are described in detail.

[0036] Figure 2 is a schematic configuration diagram of a robot according to an embodiment of the present disclosure. Figure 3 is a diagram for explaining a plurality of directional microphones according to an embodiment of the present disclosure.

[0037] The robot 100 according to an embodiment of the present disclosure may include a LiDAR sensor 110 , a plurality of directional microphones 120 , and a processor 130 .

[0038] The robot 100 may obtain information about the distance to an object (e.g., a person, an obstacle, or a wall) existing around the robot 100 by using the LiDAR sensor 110. Specifically, the LiDAR sensor 110 may be rotated 360° and irradiate laser light to an object within a predetermined radius around the robot 100, and when receiving laser light irradiated from the sensor and then reflected from the object, the distance to the object is detected based on the time at which the laser light is received. Alternatively, the LiDAR sensor 110 may detect the distance to the object by measuring the amount of phase change of the received laser light.

[0039] The LiDAR sensor 110 may include a light emitting unit that outputs laser light and a receiving unit that receives the laser light output from the light emitting unit and then reflected from an object. The object may include a person, an obstacle, a wall, an electronic device, etc. located around the robot 100.

[0040] A plurality of directional microphones (a first directional microphone 120-1, a second directional microphone 120-2, ..., and an nth directional microphone 120-n, where "n" is a natural number greater than or equal to 2) may obtain audio signals around the robot 100. Specifically, each directional microphone (a first directional microphone 120-1, a second directional microphone 120-2, ..., and an nth directional microphone 120-n, hereinafter denoted by 120) may be a microphone configured to selectively obtain only narrow-angle sounds from a specific direction.

[0041] Here, the audio signal may include not only a user's voice for controlling the robot 100 , but also various audio signals generated around the robot 100 .

[0042] According to an embodiment of the present disclosure, the directional directions of the plurality of directional microphones 120 may be set to be different from each other so as not to overlap each other. Here, the plurality of directional microphones 120 may be set in the robot 100 to correspond to different directional directions. In addition, each directional microphone may obtain an audio signal from each directional direction within a predetermined direction angle or range.

[0043] in this regard, Figure 3 It is shown that four directional microphones are provided in the robot 100 to correspond to the east, west, south, and north directions of the robot 100, respectively.

[0044] Here, the first directional microphone 120-1 provided in the robot 100 can obtain an audio signal sent in the north direction within a directional range with a width of P1. In addition, the second directional microphone 120-2 provided in the robot 100 can obtain an audio signal sent in the east direction within a directional range with a width of P2. In addition, the third directional microphone 120-3 provided in the robot 100 can obtain an audio signal sent in the south direction within a directional range with a width of P3. In addition, the fourth directional microphone 120-4 provided in the robot 100 can obtain an audio signal sent in the west direction within a directional range with a width of P4. In this way, each of the four directional microphones can obtain an audio signal sent in a different direction.

[0045] Although the directional range of each directional microphone is Figure 3 , but the embodiments of the present disclosure are not limited thereto. The plurality of directional microphones 120 may be set to different values ​​not only in the orientation direction but also in the orientation range and angle.

[0046] in addition, Figure 3 It is shown that the directivity ranges of the respective directional microphones do not overlap each other. However, based on the misalignment of the plurality of directional microphones 120 and the directivity ranges (and directivity angles) set in the directional microphones, some portions of the directivity ranges of the respective directional microphones may overlap each other.

[0047] in addition, Figure 3 Each of the plurality of directional microphones 120 is shown as a unidirectional microphone. However, the directional microphones may be implemented as bidirectional microphones, super directional microphones, and the like.

[0048] Hereinafter, for the convenience of describing the present disclosure, the robot 100 is described as including four unidirectional microphones (specifically, a first directional microphone 120 - 1 , a second directional microphone 120 - 2 , a third directional microphone 120 - 3 , and a fourth directional microphone 120 - 4 ).

[0049] The processor 130 may control the overall operation of the robot 100. Specifically, the processor 130 may control hardware or software components connected to the processor 130 by driving an operating system or an application program, and may perform various data processing and operations. In addition, the processor 130 may load instructions or data received from at least one of the other components into a volatile memory and process them, and store various data in a non-volatile memory. To this end, the processor 130 may include a central processing unit (CPU) or an application processor 130 (or an application processor (AP)).

[0050] Figure 4is an exemplary diagram illustrating obtaining information about a surrounding environment of a robot through a LiDAR sensor according to an embodiment of the present disclosure.

[0051] According to an embodiment of the present disclosure, the processor 130 may recognize objects around the robot 100 based on sensing data obtained by the LiDAR sensor 110 .

[0052] First, the processor 130 may identify whether there is an object around the robot 100 through the LiDAR sensor 110. Specifically, when receiving laser irradiated from the LiDAR sensor 110 and then reflected from the object, the processor 130 may detect the object based on the time taken from irradiation to reception of the laser.

[0053] Specifically, the processor 130 may identify the position where the laser is reflected based on the elapsed time between irradiating and receiving the reflected laser. The position where the laser is reflected may be identified as a point having a coordinate value corresponding to a position in a two-dimensional (or three-dimensional) coordinate space.

[0054] As a plurality of lasers are irradiated from the LiDAR sensor 110, the processor 130 may identify a point corresponding to a position where each laser is reflected in a two-dimensional (or three-dimensional) coordinate space. In addition, the processor 130 may identify a set of the identified plurality of points as a point cloud corresponding to the object. That is, when the laser irradiated from the LiDAR sensor 110 is reflected, the point cloud may be a set of clustered points identified on the surface of the object.

[0055] When recognizing a point cloud in a two-dimensional (or three-dimensional) coordinate space, the processor 130 may recognize that an object corresponding to the point cloud exists, and recognize a distance between the recognized object and the robot 100 .

[0056] Here, the processor 130 may identify whether there is an object within a predetermined range around the robot 100. Figure 4 , the processor 130 may be configured to only recognize objects within a radius of R value around the robot 100 .

[0057] When the time elapsed between the irradiation and the reception of the reflected laser light is too long, the position where the laser light is reflected may be located at a position far from the robot 100. Therefore, there may be an error in the coordinate value of the point corresponding to the reflected position or the point recognized by the robot 100 based on the time taken. Therefore, an error may also exist in the distance between the robot 100 and the object and the position of the object recognized by the processor 130.

[0058] Therefore, the processor 130 may recognize whether the object exists only if the recognized point cloud is within a predetermined range from the robot 100. To this end, the processor 130 may recognize a point corresponding to a position where the laser is reflected only when the elapsed time between the laser being irradiated from the LiDAR sensor 110 and the reflected laser being received again by the LiDAR sensor 110 is within a predetermined time.

[0059] Return to reference Figure 4 , the processor 130 may recognize only the wall 20-1 located north of the robot 100 and the object 10-1 located east of the object as objects around the robot 100. In addition, the processor 130 may not recognize the objects 10-2, 10-3, and 20-2 beyond the radius of the R value around the robot 100.

[0060] In an embodiment of the present disclosure, the processor 130 may recognize the type of the object based on sensing data obtained by the LiDAR sensor 110 .

[0061] Specifically, the processor 130 may identify the type of the object around the robot 100 identified based on the sensing data. The processor 130 may apply various methods to identify the type of the object based on the sensing data.

[0062] For example, the processor 130 may identify the amount of change in the distance between the object and the robot 100 based on the sensing data obtained by the LiDAR sensor 110. Here, the processor 130 may identify the amount of change in the distance between the object and the robot 100 considering the moving speed of the robot 100 when the robot 100 travels.

[0063] In addition, when the object is identified as moving or there is movement of the object based on the change amount of the distance, the processor 130 can identify the object as a person. On the other hand, when the object is identified as fixed based on the change amount of the distance, the processor 130 can identify the object as something such as an obstacle, a wall, etc.

[0064] In addition, for example, the processor 130 may identify the type of the object based on the size, width, radius, etc. of the point cloud corresponding to the object. For example, when the LiDAR sensor 110 is implemented as a three-dimensional LiDAR sensor 110, the sensing data of the object obtained by the LiDAR sensor 110 may include not only information about the distance between the robot 100 and the object, but also information about the size of the object.

[0065] Here, the processor 130 may also identify the type of the object based on information about the height of the object. When it is identified that the size of the object is greater than a predetermined size, the processor 130 may identify the object as a wall.

[0066] In addition, for example, the processor 130 may identify line data corresponding to an object having a linear shape based on distance information obtained using the LiDAR sensor 110, and identify the object based on the identified line data. Specifically, the processor 130 may extract linear components about the point cloud by using various line detection algorithms (e.g., Hough transform or HoughLines function). In addition, when the extracted linear component is equal to or greater than a predetermined size, the processor 130 may identify that the object corresponding to the extracted linear component (specifically, the object corresponding to the point cloud) is a wall.

[0067] On the other hand, when no linear component is extracted or when the extracted linear component has a size smaller than a predetermined size, the processor 130 may recognize that the object corresponding to the point cloud is not a wall. That is, for example, the processor 130 may recognize that the object corresponding to the point cloud is a person, an electronic device, furniture, etc.

[0068] Alternatively, for example, the processor 130 may convert the two-dimensional (or three-dimensional) point cloud data into a two-dimensional (or three-dimensional) image, and then input the converted two-dimensional (or three-dimensional) image into a deep learning model to identify the type of the object. Here, the deep learning model for identifying the type of the object may be a deep learning model using a conventional clustering method.

[0069] In addition, according to an embodiment of the present disclosure, the robot 100 may further include a camera (eg, a depth camera), and the processor 130 may further identify the type of the object based on an image of the object obtained by the camera and sensing data about the object obtained by the LiDAR sensor 110 .

[0070] As such, the processor 130 may recognize whether there is an object around the robot 100 , a distance between the object and the robot 100 , a type of the object, etc. by using sensing data obtained via the LiDAR sensor 110 based on various object recognition methods.

[0071] In addition, according to an embodiment of the present disclosure, the processor 130 may identify the weight of a directional microphone corresponding to the position of the recognized object among the plurality of directional microphones 120 based on the type of the recognized object when recognizing the object.

[0072] Specifically, the processor 130 may identify the type of the object and then identify the weight of the directional microphone based on the type of the object included in each directional range of the plurality of directional microphones 120 .

[0073] To this end, the processor 130 may identify objects around the robot 100 identified based on the sensing data. In addition, the processor 130 may identify objects included in the directional range and angle of each directional microphone among a plurality of objects. In addition, the processor 130 may identify the type of objects included in the directional range and angle of each directional microphone based on the sensing data. The method of identifying the type of an object is described above, and thus a detailed description thereof is omitted.

[0074] The processor 130 may identify the weight of the directional microphone corresponding to the direction range and the direction angle including each object based on the identified object type.

[0075] For example, return reference Figure 3 , assuming that it is recognized that object A exists within the directional range and directional angle of the first directional microphone 120-1, and object B exists within the directional range and directional angle of the second directional microphone 120-2. Here, the processor 130 may identify the type of object A, and then identify the weight of the first directional microphone 120-1 based on the recognized type of object A. In addition, the processor 130 may identify the type of object B, and then identify the weight of the second directional microphone 120-2 based on the recognized type of object B. In this way, the processor 130 may identify the weight of each directional microphone based on the type of object included in the directional range and directional angle of each directional microphone.

[0076] The weight of the directional microphone 120 may be a gain value applied to the audio signal obtained by the directional microphone 120. For example, assume that 2 is the weight of the recognition of the first directional microphone 120-1. Here, the processor 130 may apply the weight to the audio signal obtained by the first directional microphone 120-1, and perform recognition by doubling the strength of the audio signal obtained by the first directional microphone 120-1.

[0077] In the following, the description refers to Figures 5 to 7 Embodiments of the present disclosure are described in detail, in which weights of a plurality of directional microphones are identified based on the type of an object.

[0078] Figure 5 is an exemplary diagram illustrating weights of identifying directional microphones when an object around the robot 100 is recognized as a person according to an embodiment of the present disclosure. Figure 6 is an exemplary diagram illustrating weights of identifying directional microphones when an object around the robot 100 is recognized as a wall according to an embodiment of the present disclosure. Figure 7 is an exemplary diagram illustrating weights of identifying directional microphones when objects around the robot 100 are identified as people and walls according to an embodiment of the present disclosure.

[0079] According to an embodiment of the present disclosure, when the object is recognized as a person, the processor 130 may recognize the weight of a directional microphone in a direction corresponding to the position of the object among the plurality of directional microphones 120 as a first value.

[0080] Specifically, when the type of the object recognized based on the sensing data is recognized as a person, the processor 130 may recognize the weight of the directional microphone corresponding to the position of the object as a first value.

[0081] Specifically, the processor 130 may identify whether an object exists within the orientation angle and range of each directional microphone based on the sensing data obtained by the LiDAR sensor 110. In addition, when it is recognized that an object exists, the processor 130 may identify information about the distance between the object and the robot 100 included in the sensing data, information about a point cloud corresponding to the object, or the type of the object based on whether the object identified by the sensing data is moving, etc.

[0082] In addition, when the type of the object is identified as a person, the processor 130 may identify a directional microphone corresponding to the position of the object. That is, the processor 130 may identify a directional microphone corresponding to a directional range including the point where the person is located. In addition, the processor 130 may identify the weight of the identified directional microphone as a first value. The processor may not identify the weights of the remaining directional microphones except for the weight of the directional microphone identified as the first value, or may identify the weight as a reference value.

[0083] Reference Figure 5 , assuming that the first value is 2 and the reference value is 1, the processor 130 may identify the first value 2 as the weight of the first directional microphone 120-1 corresponding to the position of the object identified as the person 10. Here, the processor 130 may identify the reference value 1 as the weight of each of the second directional microphone 120-2 to the fourth directional microphone 120-4 corresponding to the remaining directional microphones. Therefore, the processor 130 may receive the audio signal obtained by the first directional microphone 120-1 by amplifying the gain of the audio signal by a factor of 2.

[0084] When the object is recognized as a wall, the processor 130 may recognize the weight of a directional microphone in a direction corresponding to the position of the object among the plurality of directional microphones as a second value.

[0085] Specifically, when the type of the object is identified as a wall, the processor 130 may identify a directional microphone corresponding to the position of the object and identify the second value as the weight of the identified microphone. Here, the processor 130 may identify each weight of the remaining directional microphones except the weight of the directional microphone identified as the second value as a reference value.

[0086] Reference Figure 6, assuming that the second value is 0.5 and the reference value is 1, the processor 130 may identify the second value 0.5 as the weight of the fourth directional microphone 120-4 corresponding to the position of the object identified as the wall 20. Here, the processor 130 may identify the reference value 1 as the weight of each of the first to third directional microphones 120-1 to 120-3 corresponding to the remaining directional microphones. Therefore, the processor 130 may receive the audio signal obtained by the fourth directional microphone 120-4 by reducing the gain of the audio signal by a factor of 0.5.

[0087] When it is recognized that a person and a wall simultaneously exist around the robot 100 , the processor 130 may identify weights of directional microphones corresponding to positions of respective objects recognized as the person and the wall as first values ​​and second values.

[0088] Specifically, the processor 130 may identify a directional microphone corresponding to the position of the object recognized as a person among the plurality of directional microphones 120, and identify a directional microphone corresponding to the position of the object recognized as a wall among the plurality of directional microphones 120. In addition, the processor 130 may identify the weight of the directional microphone corresponding to the position of the object recognized as a person as a first value, and identify the weight of the directional microphone corresponding to the position of the object recognized as a wall as a second value.

[0089] For example, refer to Figure 7 , the processor 130 may identify the first directional microphone 120-1 as a directional microphone corresponding to the position of the object identified as person 10. In addition, the processor 130 may identify the second directional microphone 120-2 and the fourth directional microphone 120-4 as directional microphones corresponding to the positions of the objects identified as walls 21 and 22. In addition, the processor 130 may identify the weight of the first directional microphone 120-1 as a first value, and the weights of the directional microphones of the second directional microphone 120-2 and the fourth directional microphone 120-4 as a second value. The processor 130 may identify the weight of the remaining third directional microphone 120-3 as a reference value.

[0090] According to an embodiment of the present disclosure, the first value may be greater than the second value. Specifically, the processor 130 may predict that an audio signal on which voice recognition is to be performed by the robot 100 is to be output from an object recognized as a person.

[0091] For example, the audio signal on which the robot 100 will perform voice recognition may include a voice uttered by a person including a wake-up word for the robot 100 or a command for controlling the robot 100, etc. On the other hand, when the object is recognized as a wall, the processor 130 may predict that the audio signal on which the robot 100 will perform voice recognition is not received from that direction.

[0092] Therefore, the processor 130 may apply a weight having a higher value to a directional microphone corresponding to a position of an object recognized as a person, thereby increasing a gain value of an audio signal obtained through each directional microphone.

[0093] Here, according to an embodiment of the present disclosure, when an object recognized as a person and an object recognized as a wall simultaneously exist within the direction angle and range of the same directional microphone, the processor 130 may apply a weight of a first value.

[0094] Specifically, a plurality of objects may exist within the direction angle and range of a given directional microphone. In this case, the processor 130 may identify the type of each of the plurality of objects based on the sensing data obtained by the LiDAR sensor 110. When the type of each of the plurality of objects is identified as a person and a wall, respectively, the processor 130 may identify the weight of the directional microphone corresponding to the position of the plurality of objects as a first value.

[0095] For example, return reference Figure 3 , assuming that three objects exist within the directional angle and range of the third directional microphone 120-3 among the multiple directional microphones. In addition, assuming that two of the three objects are each identified as a person, and the other is identified as a wall. Here, the processor 130 may identify the weight of the third directional microphone 120-3 as the first value instead of the second value. The reason is that although the processor 130 recognizes that there is a wall in the direction corresponding to the third directional microphone 120-3, there are two people together with the wall in the corresponding direction, so it can be predicted that verbal speech, etc. will be received in the direction corresponding to the third directional microphone 120-3.

[0096] According to an embodiment of the present disclosure, the processor 130 may identify weights of a plurality of directional microphones based on a driving state of the robot 100 .

[0097] Specifically, when it is recognized that the robot 100 is traveling, the processor 130 may respectively identify the weights of the directional microphones corresponding to the traveling direction of the robot 100 and the weights of the directional microphones corresponding to the direction opposite to the traveling direction.

[0098] Here, the directional microphone corresponding to the traveling direction may refer to a directional microphone disposed in a direction consistent with the traveling direction among the directional microphones disposed separately from each other in the robot 100. Figure 3 , when the robot 100 moves eastward, the second directional microphone 120 - 2 may be a directional microphone corresponding to the moving direction of the robot 100 .

[0099] According to an embodiment of the present disclosure, the processor 130 may identify the weight of the directional microphone corresponding to the traveling direction of the robot 100 as a third value, and identify the weight of the directional microphone corresponding to the direction opposite to the traveling direction of the robot 100 as a fourth value.

[0100] Here, the fourth value may be greater than the third value. This configuration applies a greater weight value to an audio signal received in a direction opposite to the traveling direction of the robot 100, which corresponds to a direction relatively far from the robot 100. When the robot 100 travels, the distance between the robot 100 and a user located in a direction opposite to the traveling direction of the robot 100 may increase. Here, even if the user verbally issues a control command for the robot 100, the robot 100 may obtain the user's voice despite a smaller sound. Therefore, the robot may not be able to correctly perform voice recognition on the received user's voice. Therefore, the processor 130 may obtain a clearer audio signal by increasing the strength of the audio signal that may be received in the direction opposite to the traveling direction of the robot 100.

[0101] Figure 8 is a diagram illustrating a method of identifying a preprocessing model of context information based on a robot according to an embodiment of the present disclosure.

[0102] According to an embodiment of the present disclosure, the processor 130 may obtain context information of the robot 100 based on the sensing data, and identify the pre-processing model 30 corresponding to each of the plurality of directional microphones 120 based on the context information.

[0103] Specifically, the processor 130 may identify weights of the plurality of directional microphones 120 and then identify a preprocessing model 30 to be applied to the audio signal received through the plurality of directional microphones 120. Here, the processor 130 may identify a preprocessing model for preprocessing the audio signal received through the plurality of directional microphones 120 in consideration of the surrounding environment of the robot 100.

[0104] To this end, the processor 130 may obtain context information of the robot 100 based on the sensing data. The context information may be information for the robot 100 to recognize its surrounding environment, and may include information about objects around the robot 100, spatial information about the location of the robot 100, and information about the driving state of the robot 100.

[0105] For example, the context information may include information indicating whether the space where the robot 100 is located is a closed space, whether the robot 100 is located in a narrow passage, whether there are many people around the robot 100, whether the robot 100 is traveling, etc.

[0106] The preprocessing model 30 may include a deep learning model that is trained to remove noise from an audio signal obtained by a directional microphone, restore a distorted audio signal, or restore lost information in an audio signal. For example, the preprocessing model 30 may include a deep noise autoencoder that is unsupervisedly trained to remove noise from an audio signal obtained by the robot 100 or restore lost information based on the environment of the robot 100.

[0107] More specifically, the memory of the robot 100 may store a plurality of preprocessing models 30 trained based on various environments in which the robot 100 is located, such as the driving state of the robot 100, the type of objects around the robot 100, or the distance between the robot 100 and the object. The processor 130 may recognize the surrounding environment of the robot 100 using the context information obtained based on the sensing data, and then obtain the preprocessing model 30 corresponding to the recognized environment from the memory.

[0108] Reference Figure 8 , the memory of the robot 100 may store information about a plurality of preprocessing models corresponding to respective environments as a table. Here, the processor 130 may identify whether the space where the robot 100 is located is a closed space or an open space based on the sensing data. In addition, the processor 130 may identify whether there are objects around the robot 100, the number of objects, the type of objects, the position of the objects, the distance between the objects and the robot 100, etc. based on the information about the sensing data. In addition, the processor 130 may also identify the driving state of the robot 100 based on the sensing data, that is, whether the robot is moving or stationary.

[0109] In addition, the processor 130 may identify the context information of the robot 100 by combining the information about the space, the object, and the driving state of the robot 100 identified based on the sensing data. For example, the processor 130 may obtain the first context information of the robot 100 as a condition that there is an object recognized as a person in the first direction of the robot 100 currently traveling in the open space. Here, the processor 130 may identify the preprocessing model A as a preprocessing model corresponding to the first context information. Here, the preprocessing model A may be a deep learning model that is trained to preprocess the audio signal obtained by the robot 100 through the microphone when there is an object recognized as a person in the first direction of the robot 100 traveling in the open space.

[0110] In this manner, the processor 130 may obtain context information, recognize the environment and state of the robot 100 based on the obtained context information, and recognize the environment of the robot 100 suitable for recognition and the recognized preprocessing model 30 .

[0111] The first context information according to an embodiment of the present disclosure may include information about the type of the object, information about the position of the object, information about the number of objects, and information about the distance between the robot 100 and the object. In addition, the second context information may include information about whether the robot 100 is traveling, information about the traveling speed of the robot 100, and information about the traveling direction of the robot 100.

[0112] Specifically, the first context information may be context information about the space around the robot 100 and objects present around the robot, and the second context information may be context information about the robot 100. Here, the processor 130 may obtain the first context information about the space around the robot 100 and the second context information about the driving state of the robot 100 based on the sensing data, and identify a preprocessing model corresponding to each directional microphone based on at least one of the obtained first context information and the second context information.

[0113] The preprocessing model 30 applied to each audio signal received by each directional microphone may depend on the context information. For example, assuming that the robot 100 includes four directional microphones, in the case of the first context information, the same preprocessing model A may be applied to the four audio signals received by the four directional microphones, and on the other hand, in the case of the second context information, the preprocessing model B may be applied to the two audio signals received by two of the four directional microphones, and the preprocessing model C may be applied to the two audio signals received by the remaining two directional microphones. In this way, the preprocessing model applied to each directional microphone may also depend on the surrounding environment of the robot.

[0114] Although the preprocessing model is described as being stored in the memory of the robot 100, the processor 130 may obtain the preprocessing model corresponding to the context information from the server. Specifically, when the processor 130 sends the context information of the robot 100 to the server through the communication interface, the server may identify the preprocessing model 30 corresponding to the received context information and send it to the processor 130. In addition, the processor 130 may receive the preprocessing model 30 sent by the server through the communication interface.

[0115] The processor 130 may identify the preprocessing model 30 corresponding to the plurality of directional microphones 120 and then apply the identified weights to an audio signal received through a directional microphone corresponding to the position of the identified object among the plurality of audio signals received through the plurality of directional microphones 120 .

[0116] Specifically, the processor 130 may obtain multiple audio signals through the multiple directional microphones 120. Here, as described above, each directional microphone may obtain an audio signal received from the directional direction of each directional microphone. In addition, the processor 130 may apply each recognized weight of each directional microphone to the audio signal obtained by each directional microphone.

[0117] For example, it is assumed that 3 is the weight of the recognition of the first directional microphone 120-1, 0.5 is the weight of the recognition of the second directional microphone 120-2, and there is no weight of the recognition of the third directional microphone 120-3 or the fourth directional microphone 120-4. Here, the processor 130 may apply the recognized weight 3 to the first audio signal obtained by the first directional microphone 120-1 to triple the strength (e.g., amplitude) of the first audio signal. On the other hand, the processor may apply the recognized weight 0.5 to the second audio signal obtained by the second directional microphone 120-2, thereby reducing the strength (e.g., amplitude) of the second audio signal by 0.5 times. In addition, the processor 130 may not apply weights to the third audio signal obtained by the third directional microphone 120-3 and the fourth audio signal obtained by the fourth directional microphone 120-4. In addition, the processor 130 may obtain a plurality of preprocessed audio signals by inputting the audio signals to which the weights are applied and the remaining audio signals to which the weights are not applied, respectively, into the preprocessing model set for each directional microphone.

[0118] Specifically, the processor 130 may input an audio signal having a strength different from that recognized when the audio signal is received through the microphone due to the weight applied thereto, and an audio signal having the same strength as that recognized when the audio signal is received through the microphone due to no weight applied thereto to the preprocessing model identified as corresponding to each directional microphone. Here, the preprocessing models identified as corresponding to the plurality of directional microphones 120 as described above may be the same or different from each other based on the environment of the robot 100.

[0119] For the above example, in other words, with respect to the first directional microphone 120-1, the processor 130 may input an audio signal having a strength three times that of the first audio signal received through the first directional microphone 120-1 to the preprocessing model identified as corresponding to the first directional microphone 120-1. Then, the processor 130 may obtain a first preprocessed audio signal corresponding to the first audio signal. On the other hand, with respect to the second directional microphone 120-2, the processor 130 may input an audio signal having a strength 0.5 times that of the first audio signal received through the second directional microphone 120-2 to the preprocessing model identified as corresponding to the second directional microphone 120-2. Then, the processor 130 may obtain a second preprocessed audio signal corresponding to the second audio signal. In addition, the processor 130 may obtain a third preprocessed audio signal and a fourth preprocessed audio signal by inputting the third audio signal and the fourth audio signal received through the third directional microphone 120-3 and the fourth directional microphone 120-4, respectively, as is, to the preprocessing models identified corresponding to the third directional microphone 120-3 and the fourth directional microphone 120-4, respectively.

[0120] In this way, among the multiple audio signals obtained by the multiple directional microphones 120, the strength of the audio signal may depend on whether the object exists within the directional angle and directional range of each directional microphone or the type of the object, and the type of the preprocessing model input to each audio signal may also depend on the environment of the robot 100. In this way, the processor 130 can obtain an improved and optimal audio signal for performing voice processing based on the surrounding environment of the robot.

[0121] The processor 130 may obtain a plurality of preprocessed audio signals and then perform speech recognition based on the obtained plurality of preprocessed audio signals.

[0122] Specifically, the processor 130 may extract the obtained plurality of audio signals or feature information (feature vectors) from the plurality of audio signals. In addition, the processor 130 may obtain information on speech recognition corresponding to the feature information based on an acoustic model and a language model.

[0123] Here, the information on speech recognition may include pronunciation information, phoneme information, and character string information corresponding to feature information obtained based on an acoustic model, and text data corresponding to the obtained pronunciation information based on a language model.

[0124] To this end, the memory of the robot 100 may include a speech-to-text (STT) model, an automatic speech recognition (ASR) model, a natural language understanding (NLU) model, and the like.

[0125] Here, the ASR model may include an acoustic model (AM), a pronunciation model (PM), and a language model (LM). AM may extract acoustic features of the received user voice and obtain a phoneme sequence. In addition, PM may include a pronunciation dictionary (or pronunciation dictionary), and obtain a word sequence by mapping the obtained phoneme sequence to words. In addition, LM may assign probabilities to the obtained word sequence. That is, the ASR model may obtain text corresponding to the user voice through an artificial intelligence model such as AM, PM, or LM.

[0126] The ASR model may include an end-to-end speech recognition model in which the components of AM, PM, and LM are combined with each other into a single neural network.

[0127] In addition, the processor 130 may recognize the position where the voice is uttered, that is, the position of the sound source, by using a plurality of preprocessed audio signals.

[0128] Specifically, the processor 130 may identify a time delay of an audio signal received through each of the plurality of directional microphones 120, and set a time delay for a plurality of specific positions around the robot based on the identified time delay of each directional microphone 120. That is, when receiving an audio signal output from a specific position (or distance), the processor 130 may set the time delay of the audio signal received from each directional microphone 120 to correspond to the specific position (or distance).

[0129] When receiving an audio signal through the plurality of directional microphones 120, the processor 130 may identify at least one candidate position within a predetermined range around the robot 100. Here, the candidate position may be a specific position where the audio signal received by the robot is assumed to be output among a plurality of specific positions around the robot 100. In addition, the processor 130 may obtain beamforming output power by applying a time delay set for the identified candidate position to the audio signal received through the plurality of directional microphones 120 and synthesizing the audio signals with each other.

[0130] That is, the processor 130 may obtain one audio signal by moving a plurality of audio signals by a predetermined time delay (frequency, etc.) relative to the candidate position and then synthesizing the delayed plurality of audio signals with each other. In addition, the processor 130 may calculate the beamforming output power for one obtained audio signal. In this way, for a plurality of candidate positions, the processor 130 may obtain the beamforming output power based on the time delay set for each candidate position. In addition, the processor 130 may identify the candidate position where the beamforming output power is maximized as the position of the sound source. The processor 130 may use a steering response power with phase transformation (SRP-PHAT) algorithm to identify the position of the sound source.

[0131] The predetermined time delay for the candidate position may be a value set in consideration of directions in which the plurality of directional microphones 120 are disposed and distances between the plurality of directional microphones 120 to calculate the highest beamforming power for the precise position of the actual sound source.

[0132] The processor 130 may identify the position of the sound source using a subspace estimation method, a time delay estimation method, or the like.

[0133] Hereinafter, each type of preprocessing model applied to a plurality of directional microphones based on the surrounding environment of the robot 100 and embodiments of the present disclosure related thereto are described.

[0134] Fig. 9 is an exemplary diagram illustrating recognition of a preprocessing model applied to a plurality of directional microphones when a robot is recognized as traveling according to an embodiment of the present disclosure.

[0135] Fig.10 is an exemplary diagram illustrating recognition of a preprocessing model applied to a plurality of directional microphones when a robot is recognized as moving toward a person according to an embodiment of the present disclosure.

[0136] According to an embodiment of the present disclosure, when the processor 130 recognizes that the robot 100 is traveling based on the second context information, the pre-processing model corresponding to the plurality of directional microphones may include the first model.

[0137] The processor 130 may recognize that the robot 100 is traveling based on the second context information about the driving state of the robot 100. Specifically, the processor 130 may detect a change in the relative distance between the robot 100 and an object around the robot 100 based on the sensing data obtained by the LiDAR sensor 110. The processor 130 may recognize whether the robot 100 is traveling and the traveling direction of the robot 100 through the detected change in the relative distance between the robot 100 and the object around the robot 100.

[0138] When the robot 100 is traveling, noise may occur due to friction noise occurring from the robot 100 contacting the ground, road, etc. on which the robot 100 is located.

[0139] The first model 30-1 according to an embodiment of the present disclosure may be a preprocessing model trained to remove or reduce noise occurring from a road or the like while the robot 100 is traveling. To this end, the first model 30-1 may be pre-learned by the robot 100 through first audio data including noise and second audio data not including noise.

[0140] When recognizing that the robot 100 is traveling based on the sensing data, the processor 130 may recognize that the preprocessing model for the plurality of directional microphones 120 is a preprocessing model including the first model 30 - 1 .

[0141] Specifically, the processor 130 may equally input a plurality of audio signals obtained through a plurality of directional microphones to the preprocessing model including the first model 30-1. In addition, the processor 130 may obtain a plurality of preprocessed audio signals respectively corresponding to the audio signals as output data of the preprocessing model including the first model 30-1.

[0142] For example, refer to Figure 3 and Fig. 9 , the processor 130 may recognize that the robot 100 is traveling in the north direction based on the acquired sensing data. Here, the processor 130 may input all the first to fourth audio signals obtained from the first to fourth directional microphones 120-1 to 120-4 into the preprocessing model including the first model 30-1. In addition, the processor 130 may obtain a plurality of preprocessed audio signals respectively corresponding to the first to fourth audio signals as output data of the preprocessing model including the first model 30-1.

[0143] The size and waveform of the noise occurring between the robot 100 and the ground depend on the traveling speed of the robot 100. Therefore, according to an embodiment of the present disclosure, the first model 30-1 may include a plurality of pre-processing models corresponding to a plurality of levels classified based on the traveling speed of the robot 100. That is, the first model 30-1 may include a plurality of first pre-processing models trained to remove noise that occurs differently based on the traveling speed of the robot 100.

[0144] To this end, a plurality of levels may be set based on a predetermined range of travel speeds. For example, the first level may be set to a travel speed of the robot 100 of zero km / h or more and less than 1 km / h, the second level may be set to a travel speed of 1 km / h or more and less than 2 km / h, and the third level may be set to a travel speed of 2 km / h or more and less than 3 km / h. In addition, the first model may include a plurality of preprocessing models (e.g., a 1-1 model corresponding to the first level, a 1-2 model corresponding to the second level, and a 1-3 model corresponding to the third level) trained to remove noise occurring between the ground and the robot 100 occurring at a travel speed corresponding to each level.

[0145] Therefore, when recognizing that the robot 100 is traveling, the processor 130 may recognize the traveling speed of the robot 100 and then recognize the level corresponding to the traveling speed of the robot 100. In addition, the processor 130 may apply a preprocessing model including a first model corresponding to the recognized traveling speed to the plurality of directional microphones.

[0146] In addition, according to an embodiment of the present disclosure, when it is identified based on the first contextual information and the second contextual information that the robot 100 is moving toward an object identified as a person, the processor 130 may identify the preprocessing model including the first model 30-1 and the second model 30-2 as a preprocessing model corresponding to the directional microphone in the direction corresponding to the position of the object.

[0147] The second model 30-2 according to an embodiment of the present disclosure may be a preprocessing model that is trained to prevent the audio signal output from the sound source from being distorted due to a change in the distance between the traveling robot 100 and the sound source that outputs the audio signal, or to restore information lost in the audio signal. To this end, the second model 30-2 may be pre-trained for the same text using third audio data obtained when the robot 100 is moving and fourth audio data obtained when the robot 100 is stationary.

[0148] The processor 130 may recognize that the robot 100 is traveling based on the second context information, and recognize that the robot 100 is traveling toward an object recognized as a person based on the first context information and the second context information.

[0149] Specifically, the processor 130 may obtain sensing data about the surrounding environment of the robot 100 that changes as the robot 100 moves in real time through the LiDAR sensor 110. Here, the processor 130 may recognize that an object exists within the directional angle and range of the directional microphone corresponding to the direction in which the robot 100 moves based on the sensing data obtained through the LiDAR sensor 110. In addition, when the type of the object is recognized as a person based on the sensing data, the processor 130 may recognize that the robot 100 is moving toward the person.

[0150] Alternatively, the processor 130 may identify an object around the robot 100 when the robot 100 is stationary based on the first context information. Here, when the robot 100 starts to travel based on the received travel command, the processor 130 may identify the travel direction of the robot 100 based on the second context information. In addition, the processor 130 may recognize that the robot 100 is traveling toward the object recognized as a person when recognizing that the travel direction of the robot 100 corresponds to the position of the object recognized as a person based on the first context information and the second context information.

[0151] In this way, when the robot 100 is identified as moving toward an object identified as a person based on the first contextual information and the second contextual information, the processor 130 can identify the preprocessed model including the first model 30-1 and the second model 30-2 as a preprocessed model corresponding to the directional microphone in the direction corresponding to the position of the object.

[0152] That is, the processor 130 may connect the second model trained to restore distortion of the audio signal occurring due to a change in the distance between the robot 100 and a person to the first model 30 - 1 applied when the robot 100 is recognized as traveling.

[0153] Here, according to an embodiment of the present disclosure, the processor 130 may apply a weight of a first value to an audio signal received by a directional microphone in a direction corresponding to the position of an object recognized as a person, obtain a first preprocessed audio signal by inputting the audio signal to which the weight of the first value is applied to the first model 30-1, and obtain a second preprocessed audio signal by inputting the obtained first preprocessed audio signal to the second model 30-2. In addition, the processor 130 may not apply a weight to an audio signal (or apply a reference weight thereto), the audio signal being received by the remaining directional microphones among a plurality of directional microphones except for the directional microphone to which the weight of the first value is applied, and obtain a preprocessed audio signal for each audio signal by inputting the received audio signal to the first model 30-1.

[0154] Reference Fig.10, the processor 130 may recognize that the robot 100 is traveling in the north direction based on the obtained second context information. Here, the processor 130 may recognize the preprocessing model corresponding to the first microphone 120-1 to the fourth microphone 120-4 as a preprocessing model including the first model 30-1. In addition, the processor 130 may recognize that the robot 100 is traveling toward an object identified as a person based on the first context information and the second context information. Here, the processor 130 may recognize that the preprocessing model also includes a second model 30-2, which corresponds to the first directional microphone 120-1 corresponding to the directional microphone of the north direction, wherein the north direction corresponds to the position of the object 10 identified as a person. Therefore, the processor 130 may apply a weight of a first value to a first audio signal obtained by the first directional microphone 120-1, obtain a first preprocessed audio signal by inputting the first audio signal to which the weight of the first value is applied to the first model 30-1, and obtain a second preprocessed audio signal by inputting the obtained first preprocessed audio signal to the second model 30-2. The processor 130 may not apply weights to the second to fourth audio signals obtained through the second to fourth directional microphones 120-2 to 120-4, and obtain each preprocessed audio signal by inputting the second to fourth audio signals to the first model 30-1.

[0155] In addition, according to an embodiment of the present disclosure, when the robot 100 is identified as being adjacent to an object identified as a wall based on the first context information, the processor 130 may identify the preprocessing model including the third model 30-3 as a preprocessing model corresponding to a plurality of directional microphones. That is, when it is identified that the robot 100 is located in a closed space surrounded by walls or is traveling through a narrow passage, the processor 130 may identify the preprocessing model including the third model as a preprocessing model corresponding to a plurality of directional microphones 120 of the robot 100.

[0156] Here, the third model 30-3 may be a deep learning-based preprocessing model trained to remove audio signals (i.e., echo noise) reflected from walls. Specifically, the third model 30-3 may be a model trained to remove echo noise included in an audio signal obtained by a directional microphone. To this end, the third model 30-3 may be pre-trained for a plurality of texts using fifth audio data not including echo noise and sixth audio data including echo noise.

[0157] Specifically, the processor 130 may identify the distance between the object identified as a wall around the robot 100 and the robot 100 based on the first context information. In addition, when it is identified that the distance between the object identified as a wall and the robot 100 is within a predetermined distance, the processor 130 may recognize that the robot 100 and the object identified as a wall are adjacent to each other.

[0158] In an embodiment of the present disclosure, the processor 130 may recognize that the robot 100 is adjacent to an object identified as a wall based on line data corresponding to the object. Specifically, when it is recognized that the size of the line data corresponding to the object is a predetermined size or larger, the processor 130 may recognize that the object is a wall. Here, when it is recognized that a plurality of line data corresponding to the object identified as a wall intersect with each other and the distance between the robot 100 and the intersection of the line data is within a predetermined distance, the processor 130 may recognize that the robot 100 is adjacent to the wall. Optionally, the processor 130 may also recognize that the robot 100 is adjacent to the wall when the minimum distance between the robot 100 and each line data is recognized and the recognized minimum distances are all within a predetermined distance.

[0159] In this way, when it is recognized that the robot 100 is traveling adjacent to an object recognized as a wall, the processor 130 may recognize the preprocessing model corresponding to the plurality of directional microphones as a preprocessing model including the third model 30-3. In this way, the processor 130 may obtain a preprocessed audio signal from which echo noise is removed from the received audio signal.

[0160] Fig.11 is an exemplary diagram illustrating a preprocessing model applied to a plurality of directional microphones when a robot is recognized as traveling toward a person adjacent to a wall according to an embodiment of the present disclosure.

[0161] According to an embodiment of the present disclosure, when it is identified based on the first contextual information and the second contextual information that the robot 100 is moving toward an object identified as a wall, the processor 130 may identify a preprocessed model including a first model 30-1, a second model 30-2, and a third model 30-3 as a preprocessed model corresponding to a directional microphone in a direction corresponding to the position of the object.

[0162] Here, according to an embodiment of the present disclosure, the processor 130 may apply a weight of a first value to an audio signal received through a directional microphone in a direction corresponding to the position of an object identified as a person, and apply a weight of a second value to an audio signal received through a directional microphone in a direction corresponding to the position of an object identified as a wall.

[0163] In addition, the processor 130 can obtain a first preprocessed audio signal by inputting an audio signal to which a weight of a first value is applied to the first model 30-1, obtain a second preprocessed audio signal by inputting the obtained first preprocessed audio signal to the second model 30-2, and obtain a third preprocessed audio signal by inputting the obtained second preprocessed audio signal to the third model 30-3.

[0164] In addition, the processor 130 may obtain a fourth preprocessed audio signal by inputting the audio signal to which the weight of the second value is applied to the first model 30-1, and obtain a fifth preprocessed signal by inputting the obtained fourth preprocessed audio signal to the third model 30-3.

[0165] Here, the processor 130 may obtain a sixth preprocessed audio signal by inputting the remaining audio signal to which the weight of the first value or the second value is not applied to the first model 30-1, and obtain a seventh preprocessed signal by inputting the obtained sixth preprocessed audio signal to the third model 30-3.

[0166] This description describes the preprocessing models to which the audio signal is input in the order of the first model to the third model, but the embodiments of the present disclosure are not limited thereto. That is, the order of the first model to the third model may vary depending on a given embodiment.

[0167] Specifically, refer to Fig.11 , the processor 130 may recognize that the robot 100 is traveling adjacent to a wall based on the first contextual information and the second contextual information. In addition, the processor 130 may recognize that the preprocessing models corresponding to the first microphone 120-1 to the fourth microphone 120-4 also include a third model 30-3. Therefore, the processor 130 may apply a weight of a first value to the first audio signal obtained by the first directional microphone 120-1, obtain a 1-1 preprocessed audio signal by inputting the first audio signal to which the weight of the first value is applied to the first model 30-1, obtain a 1-2 preprocessed audio signal by inputting the obtained 1-1 preprocessed audio signal to the third model 30-3, and obtain a 1-3 preprocessed audio signal by inputting the obtained 1-2 preprocessed audio signal to the second model 30-2.

[0168] The processor 130 may apply the weight of the second value to the second audio signal and the fourth audio signal obtained by the second directional microphone 120-2 and the fourth directional microphone 120-4 in the direction corresponding to the position of the object recognized as the wall. In addition, the processor 130 may obtain the 2-1st preprocessed audio signal and the 4-1st preprocessed audio signal, which are preprocessed respectively, by inputting the second audio signal and the fourth audio signal to which the weight of the second value is applied to the first model 30-1. In addition, the processor 130 may obtain each of the 2-2nd preprocessed audio signal and the 4-2nd preprocessed audio signal by inputting the 2-1st preprocessed audio signal and the 4-1st preprocessed audio signal to the third model 30-3.

[0169] In addition, the processor 130 may not apply a weight to the third audio signal obtained by the third directional microphone 120-3, and obtain a 3-1st preprocessed audio signal by inputting the third audio signal to the first model 30-1. In addition, the processor 130 may obtain a 3-2nd preprocessed audio signal by inputting the 3-1st preprocessed audio signal to the third model 30-3.

[0170] According to an embodiment of the present disclosure, when it is recognized based on the first context information that a plurality of objects recognized as humans exist around the robot 100, the processor 130 may recognize the preprocessing model including the fourth model as a preprocessing model corresponding to the plurality of directional microphones.

[0171] The fourth model may be a deep learning-based preprocessing model trained to remove babble noise. Specifically, the fourth model may be model-trained to remove non-stationary type noise such as babble noise (such as humming) included in the audio signal obtained by the directional microphone when the object is identified as many people. To this end, the fourth model may be pre-trained for multiple texts by the seventh audio data not including noise and the eighth audio data including babble noise.

[0172] The processor 130 may identify information about the space around the robot 100 through the first context information. Specifically, when multiple objects around the robot 100 are identified and the multiple objects are people, the processor 130 may identify the fourth model as a preprocessing model. Here, the processor 130 may identify the fourth model as a preprocessing model corresponding to multiple directional microphones. That is, the processor 130 may apply the fourth model to multiple audio signals obtained through the multiple directional microphones in the same manner.

[0173] However, according to an embodiment of the present disclosure, when the distance between the robot 100 and the object recognized as a person is above a predetermined distance, the processor 130 may apply the fourth model only to the directional microphones in the direction corresponding to the position of the object recognized as a person among the plurality of directional microphones. Specifically, the processor 130 may cluster the plurality of objects recognized as people by using the sensing data. In addition, the processor 130 may identify the distance between the group including the clustered plurality of objects and the robot 100.

[0174] Here, the processor 130 may identify the distance between each object included in the group and the robot 100 based on the sensing data, and then calculate the average value of the identified distances. In addition, when the calculated average value is compared with the predetermined distance and then it is identified that the calculated average value is the predetermined distance or more, the processor 130 may identify the preprocessing model for the directional microphone in the direction corresponding to the position of the group including the plurality of objects among the plurality of directional microphones as the fourth model.

[0175] Fig.12 is a detailed configuration diagram of a robot according to an embodiment of the present disclosure.

[0176] Reference Fig.12 According to an embodiment of the present disclosure, the robot 100 may include a LiDAR sensor 110, a plurality of directional microphones 120, a drive unit 140, a memory 150, a display 160, an input / output interface 170, a communication interface 180, and a processor 130. This description omits Fig.11 The components shown are Figure 2 The components shown overlap the detailed description of the components.

[0177] The driving unit 140 is a component for moving the robot 100. The driving unit 140 may be implemented as a wheel or a leg of the robot 100. To this end, the driving unit 140 may include a motor. The processor 130 may control various travel operations such as movement, stop, speed control, and direction change of the robot 100 by controlling the driving unit 140.

[0178] The memory 150 may store data necessary for various embodiments of the present disclosure. Specifically, the memory 150 may store a preprocessing model corresponding to the context information of the robot according to an embodiment of the present disclosure. For example, the memory 150 may be based on Figure 8 The context information of the robot shown in stores information about the preprocessing model in the form of a table.

[0179] The memory 150 may be implemented in the form of a memory 150 embedded in the robot 100 or in the form of a memory 150 detachable from the robot 100 based on the purpose of data storage. For example, the memory 150 embedded in the robot 100 may store data for driving the robot 100, and the memory 150 detachable from the robot 100 may store data for expanding functions of the robot 100.

[0180] The memory 150 embedded in the robot 100 may be implemented as at least one of a volatile memory (e.g., a dynamic random access memory (DRAM), a static RAM (SRAM), or a synchronous dynamic RAM (SDRAM)), a nonvolatile memory (e.g., a one-time programmable read-only memory (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, or a flash ROM), a flash memory (e.g., a NAND flash memory or a NOR flash memory), a hard disk drive, or a solid state drive (SSD)).

[0181] In addition, the memory 150 removable from the robot 100 may be implemented in the form of a memory card (e.g., Compact Flash (CF), Secure Digital (SD), Micro Secure Digital (Micro-SD), Mini Secure Digital (Mini-SD), Extreme Digital (xD), or MultiMedia Card (MMC)) or an external memory connectable to a Universal Serial Bus (USB) port (e.g., USB memory 1).

[0182] The display 160 may display various data information. For example, the processor 130 may display information about the travel path of the robot 100 through the display 160. The display 160 may be implemented as various types of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, and a plasma display panel (PDP). The display 160 may also include a drive circuit, a backlight unit, and the like that may be implemented in a form such as a thin film transistor (TFT), a low temperature polysilicon (LTPS) TFT, or an organic TFT (OTFT). The display 160 may be implemented as a flexible display, a three-dimensional (3D) display, and the like.

[0183] The input / output interface 170 is a component for the robot 100 to interact with a user, and the processor 130 may receive various information through the input / output interface 170. The input / output interface 170 may include at least one of a touch sensor, a motion sensor, a button, a jog dial, a switch, or a speaker, but is not limited thereto.

[0184] The communication interface 180 may input and output various types of data. Specifically, the processor 130 may communicate with an external server through the communication interface 180. Here, the processor 130 may send the context information of the robot to the external server through the communication interface 180. In addition, the processor 130 may receive information about a preprocessing model corresponding to the context information of the robot from the external server through the communication interface 180.

[0185] To this end, the communication interface 180 may use a communication method such as access point (AP)-based Wireless Fidelity (Wi-Fi, i.e., wireless local area network (LAN)), Bluetooth, Zigbee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE 1394, High-Definition Multimedia Interface (HDMI), USB, Mobile High-Definition Link (MHL), Audio Engineering Society / European Broadcasting Union (AES / EBU) communication, optical communication, or coaxial communication.

[0186] Fig.13 is a flowchart schematically illustrating a control method of the robot 100 according to another embodiment of the present disclosure.

[0187] Reference Fig.13 According to another embodiment of the present disclosure, the processor 130 may recognize objects around the robot 100 based on sensing data obtained by the LiDAR sensor 110 (S1310).

[0188] Specifically, the processor 130 may identify whether there is an object around the robot 100 and the distance between the object and the robot based on the time it takes for the laser irradiated by the LiDAR sensor 110 to be received by the LiDAR sensor 110 again.

[0189] Then, the processor 130 may identify a weight of a directional microphone corresponding to the position of the recognized object among the plurality of directional microphones based on the type of the object recognized when the object is recognized ( S1320 ).

[0190] Specifically, the processor 130 can identify whether there is an object within the directional range and angle of each directional microphone, and can identify the type of the identified object based on the sensing data. In addition, the processor 130 can identify the weight of the directional microphone corresponding to the directional range and angle of the position including the identified object among the multiple directional microphones based on the type of the identified object.

[0191] Then, the processor 130 may obtain context information of the robot 100 based on the sensing data, and identify a pre-processing model corresponding to each of the plurality of directional microphones based on the context information ( S1330 ).

[0192] Specifically, the processor 130 may identify a preprocessing model trained to preprocess an audio signal received through a microphone based on the surrounding environment of the robot 100. To this end, the processor 130 may obtain context information of the robot 100 based on the sensing data, and identify the surrounding environment of the robot 100 based on the obtained context information. Here, the surrounding environment of the robot 100 may involve the type of space where the robot 100 is located, the type of objects around the robot 100, the distance between the robot 100 and the object, the travel state of the robot 100, and the like.

[0193] The processor 130 may recognize the surrounding environment of the robot 100 and then recognize a preprocessing model corresponding to the recognized environment. Here, the processor 130 may recognize the preprocessing models recognized for the plurality of directional microphones as being the same or different from each other based on the environment.

[0194] The processor 130 may then apply the identified weights to audio signals received by a directional microphone corresponding to the position of the identified object among a plurality of audio signals received by a plurality of directional microphones (S1340), and obtain a plurality of preprocessed audio signals by inputting the audio signals to which the weights are applied and the remaining audio signals into a preprocessing model corresponding to each directional microphone (S1350).

[0195] In this way, the processor 130 may obtain an improved audio signal by appropriately pre-processing the audio signal that is distorted by the surrounding environment of the robot 100 or includes some lost information.

[0196] Then, the processor 130 may perform voice recognition based on the obtained plurality of pre-processed audio signals ( S1360 ).

[0197] In the above detailed description, according to another embodiment of the present disclosure, operations S1310 to S1360 may be further divided into additional operations or combined into fewer operations. In addition, some operations may be omitted as needed, and the order between operations may be changed.

[0198] The methods according to the various embodiments of the present disclosure described above may be implemented in the form of an application that may be installed in a conventional robot.

[0199] In addition, the methods according to the various embodiments of the present disclosure described above may be implemented only through software upgrade or hardware upgrade of a conventional robot.

[0200] In addition, the various embodiments of the present disclosure described above may be performed by an embedded server provided in the robot or at least one external server of the robot.

[0201] The various embodiments described above can be implemented in a computer or computer-readable recording medium using software, hardware, or a combination of software and hardware. In some cases, the embodiments described in this disclosure can be implemented by the processor itself. According to software implementation, embodiments such as the processes and functions described in this disclosure can be implemented by separate software modules.

[0202] Each of the software modules may execute one or more functions and operations described in the specification.

[0203] The non-transitory computer-readable medium may store computer instructions for executing the processing operations of the robot 100 according to the various embodiments of the present disclosure described above. When the computer instructions are executed by a processor of a specific device, the computer instructions stored in the non-transitory computer-readable medium may allow the specific device to execute the processing operations of the robot 100 according to the various embodiments described above.

[0204] A non-transitory computer-readable medium is not a medium in which data is stored for a period of time, such as a register, a cache, or a memory, and indicates a medium in which data is stored semi-permanently and can be read by a machine. Specific examples of non-transitory computer-readable media may include a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a universal serial bus (USB), a memory card, a read-only memory (ROM), etc.

[0205] Although specific embodiments are shown and described in the present disclosure as above, the embodiments of the present disclosure are not limited to the above specific embodiments, and various modifications may be made by those skilled in the art to which the present disclosure belongs without departing from the subject matter of the present disclosure claimed in the appended claims. These modifications should also be understood to fall within the scope and spirit of the present disclosure.

Claims

1. A robot comprising: Light detection and ranging LiDAR sensors; Multiple directional microphones; as well as At least one processor configured to: Based on the sensing data obtained by the LiDAR sensor, identifying objects near the robot, identifying, based on the type of the object, a weight to be applied to an audio signal received by a directional microphone among the plurality of directional microphones corresponding to the location of the object, obtaining context information of the robot based on the sensing data, identifying a preprocessing model corresponding to each of the plurality of directional microphones based on the context information, applying the weight to an audio signal received by a directional microphone corresponding to the position of the object among a plurality of audio signals received by the plurality of directional microphones, obtaining a plurality of preprocessed audio signals by inputting the audio signal to which the weight has been applied and the remaining audio signals into preprocessing models corresponding to the respective directional microphones, and Speech recognition is performed based on the plurality of preprocessed audio signals.

2. The robot according to claim 1, wherein: The at least one processor is further configured to: Based on identifying the object as a person, identifying a first value as the weight applied to an audio signal received through a directional microphone corresponding to the position of the object, and Based on identifying the object as a wall, identifying a second value as the weight applied to an audio signal received through a directional microphone corresponding to the location of the object, and The first value is greater than the second value.

3. The robot according to claim 2, wherein: The at least one processor is further configured to: obtaining first context information about a space near the robot and second context information about a driving state of the robot based on the sensing data; as well as A pre-processing model corresponding to each directional microphone of the plurality of directional microphones is identified based on at least one of the first context information and the second context information.

4. The robot according to claim 3, wherein: The first context information includes at least one of information about the type of the object, information about the location of the object, information about the number of objects near the robot, and information about the distance between the robot and the object, and The second context information includes at least one of information about whether the robot is moving, information about the moving speed of the robot, and information about the moving direction of the robot.

5. The robot according to claim 3, wherein: The at least one processor is further configured to: recognizing that the robot is moving based on the second context information, and Based on recognizing that the robot is moving, a preprocessing model including a first model is identified as a preprocessing model corresponding to each of the plurality of directional microphones.

6. The robot according to claim 5, wherein: The at least one processor is further configured to: Based on recognizing that the robot is moving toward the object recognized as a person according to the first context information and the second context information, identifying the preprocessing model including the first model and the second model as a preprocessing model corresponding to a directional microphone, wherein the directional microphone corresponds to the position of the object, obtaining a first preprocessed audio signal by inputting the audio signal to which the weight of the first value is applied into a first model, and A second preprocessed audio signal is obtained by inputting the first preprocessed audio signal into a second model.

7. The robot according to claim 3, wherein: The at least one processor is further configured to: Based on identifying the object as a wall adjacent to the robot according to the first context information, a preprocessing model including a third model is identified as a preprocessing model corresponding to each of the plurality of directional microphones.

8. The robot according to claim 3, wherein: The at least one processor is further configured to: Based on identifying a plurality of objects near the robot according to the first context information and the plurality of objects being a plurality of persons, a preprocessing model including a fourth model is identified as a preprocessing model corresponding to each of the plurality of directional microphones.

9. A method for controlling a robot, the method comprising: identifying objects in the vicinity of the robot based on sensing data obtained by a light detection and ranging LiDAR sensor; identifying, based on the type of the object, a weight to be applied to an audio signal received by a directional microphone among a plurality of directional microphones that corresponds to a location of the object; Obtaining context information of the robot based on the sensing data; identifying a preprocessing model corresponding to each respective directional microphone of the plurality of directional microphones based on the contextual information; applying the weight to an audio signal received through a directional microphone corresponding to the position of the object among a plurality of audio signals received through the plurality of directional microphones; Obtaining a plurality of preprocessed audio signals by inputting the audio signal to which the weights have been applied and the remaining audio signals into preprocessing models corresponding to the respective directional microphones; as well as Speech recognition is performed based on the plurality of preprocessed audio signals.

10. The method according to claim 9, wherein: Identifying the weights further comprises: Based on identifying the object as a person, identifying a first value as the weight applied to an audio signal received through a directional microphone corresponding to the position of the object; and Based on identifying the object as a wall, identifying a second value as the weight applied to an audio signal received through a directional microphone corresponding to the location of the object, and The first value is greater than the second value.

11. The method according to claim 10, wherein: Obtaining context information further includes: obtaining first context information about a space near the robot and second context information about a driving state of the robot based on the sensing data, and The method further includes: identifying a preprocessing model corresponding to each directional microphone of the plurality of directional microphones based on at least one of the first context information and the second context information.

12. The method according to claim 11, wherein: The first context information includes at least one of information about the type of the object, information about the location of the object, information about the number of objects near the robot, and information about the distance between the robot and the object, and The second context information includes at least one of information about whether the robot is moving, information about the moving speed of the robot, and information about the moving direction of the robot.

13. The method according to claim 12, in, The recognition preprocessing model also includes, recognizing that the robot is moving based on the second context information, and Based on recognizing that the robot is moving, a preprocessing model including a first model is identified as a preprocessing model corresponding to each of the plurality of directional microphones.

14. The method according to claim 13, wherein: The recognition preprocessing model also includes: Based on recognizing that the robot is moving toward the object recognized as a person according to the first context information and the second context information, recognizing the preprocessing model including the first model and the second model as a preprocessing model corresponding to a directional microphone, wherein the directional microphone corresponds to the position of the object, and Wherein, obtaining the plurality of preprocessed audio signals further comprises: obtaining a first preprocessed audio signal by inputting the audio signal to which the weight of the first value is applied into a first model; and A second preprocessed audio signal is obtained by inputting the first preprocessed audio signal into a second model.

15. The method according to claim 11, wherein: The recognition preprocessing model also includes: Based on identifying the object as a wall adjacent to the robot according to the first context information, a preprocessing model including a third model is identified as a preprocessing model corresponding to each of the plurality of directional microphones.

Images