Robot for projecting images and image projection method thereof

Abstract

This robot comprises: a projector; a sensor; a memory storing instructions; and at least one processor. The instructions, when executed by at least one processor, may cause the robot to: identify a plurality of candidate projection areas on the basis of first information obtained by sensing the surroundings of a user via a sensor; identify priorities of the plurality of candidate projection areas; project, on the basis of the priorities and a plurality of positions of the plurality of candidate projection areas, a first image to an area including the plurality of candidate projection areas; control the projector to display second information about the priorities on the plurality of candidate projection areas; identify, as a projection area, a candidate projection area selected on the basis of a user input from among the plurality of candidate projection areas; and project image content onto the projection area via the projector.

Description

Robot for projecting images and its image projection method

[0001] The present disclosure relates to a robot for projecting an image and an image projection method thereof.

[0002] In addition to simple repetitive functions, robots can sense their surroundings in real time using sensors, cameras, and other tools, collect information, and drive autonomously.

[0003] These robots are currently being used in many fields, providing various services through interaction with users.

[0004] A robot according to an embodiment of the present disclosure includes a projector, a sensor, a memory storing instructions, and one or more processors. The instructions, when executed by the at least one processor, cause the robot to identify a plurality of candidate projection areas based on first information acquired by sensing the user's surroundings through the sensor, identify priorities of the plurality of candidate projection areas, control the projector to project a first image on an area including the plurality of candidate projection areas based on a plurality of locations of the plurality of candidate projection areas and the priorities, and display second information regarding the priorities on the plurality of candidate projection areas, identify a candidate projection area selected from among the plurality of candidate projection areas based on a user input as a projection area, and project image content on the projection area through the projector.

[0005] Additionally, the one or more processors may execute the instructions to cause the robot to identify a plurality of areas corresponding to the plurality of candidate projection areas in a second image to be projected on the area based on the plurality of positions, and to control the projector to project the second image. The second image may include a plurality of sub-images corresponding to the plurality of areas, and the plurality of sub-images may include a plurality of indicators corresponding to the priorities.

[0006] Additionally, the plurality of indicators may include a plurality of numbers indicating the priorities.

[0007] Additionally, the one or more processors may execute the instructions to cause the robot to identify a plurality of areas corresponding to the plurality of candidate projection areas based on the plurality of positions, and control the projector to sequentially project a plurality of sub-images corresponding to the plurality of areas. The plurality of sub-images may include a plurality of indicators corresponding to the priorities.

[0008] Additionally, the plurality of indicators may include a plurality of numbers indicating the priorities.

[0009] Additionally, the one or more processors may execute the instructions to cause the robot to identify the priority based on a plurality of sizes of the plurality of candidate projection areas and a plurality of distances between the user and the plurality of candidate projection areas.

[0010] Additionally, the one or more processors may execute the instructions to cause the robot to generate a three-dimensional map of the surroundings based on the first information, identify a plane in the surroundings based on the three-dimensional map, identify a plurality of areas having a first aspect ratio matching a second aspect ratio of a projection image on the plane, and identify a plurality of candidate projection areas among the plurality of areas based on characteristics of the plurality of areas.

[0011] Additionally, the one or more processors may execute the instructions to cause the robot to identify, among the plurality of regions, regions other than the identified region as candidate projection regions. The saturation of the identified region may be greater than or equal to a preset threshold, and the identified region may be determined based on RGB values ​​of a plurality of points in the plurality of regions.

[0012] Additionally, the one or more processors may execute the instructions to cause the robot to obtain a third image through the sensor, identify a location of the user from the third image, identify a rotation angle range of the sensor based on the location of the user and a field of view of the sensor, and obtain the first information through the sensor while the sensor rotates within the rotation angle range.

[0013] Additionally, the one or more processors may execute the instructions to cause the robot to obtain a bounding box for the user based on the third image, identify a pixel distance between a center pixel of the third image and a pixel of the bounding box, and identify a rotation angle range based on a focal length of the sensor and the pixel distance.

[0014] An image projection method of a robot including a projector according to an embodiment of the present disclosure includes a step of identifying a plurality of candidate projection areas based on first information acquired by sensing the user's surroundings through a sensor, a step of identifying priorities of the plurality of candidate projection areas, a step of projecting a first image through the projector in an area including the plurality of candidate projection areas based on a plurality of locations of the plurality of candidate projection areas and the priorities, a step of displaying second information regarding the priorities in the plurality of candidate projection areas, a step of identifying a candidate projection area selected from among the plurality of candidate projection areas based on a user input as a projection area, and a step of projecting image content in the projection area through the projector.

[0015] The displaying step may include a step of identifying a plurality of areas corresponding to the plurality of candidate projection areas in a second image to be projected onto the area based on the plurality of positions, and a step of controlling the projector to project the second image. The second image may include a plurality of sub-images corresponding to the plurality of areas, and the plurality of sub-images may include a plurality of indicators corresponding to the priority.

[0016] The above plurality of indicators may include a plurality of numbers indicating the priority.

[0017] The displaying step may include a step of identifying a plurality of areas corresponding to the plurality of candidate projection areas based on the plurality of positions, and a step of controlling the projector to sequentially project a plurality of sub-images corresponding to the plurality of areas. The plurality of sub-images may include a plurality of indicators corresponding to the priorities.

[0018] The above plurality of indicators may include a plurality of numbers indicating the priority.

[0019] The step of identifying the priority may include the step of identifying the priority based on a plurality of sizes of the plurality of candidate projection areas and a plurality of distances between the user and the plurality of candidate projection areas.

[0020] The step of identifying the plurality of candidate projection areas may include the steps of generating a three-dimensional map of the surroundings based on the first information, identifying a plane in the surroundings based on the three-dimensional map, identifying a plurality of areas having a first aspect ratio matching a second aspect ratio of a projection image on the plane, and identifying the plurality of candidate projection areas among the plurality of areas based on characteristics of the plurality of areas.

[0021] The step of identifying the plurality of candidate projection areas may include a step of identifying areas remaining from the plurality of areas, excluding the identified areas, as the plurality of candidate projection areas. The saturation of the identified areas may be greater than or equal to a preset threshold value, and the identified areas may be determined based on RGB values ​​of a plurality of points in the plurality of areas.

[0022] The step of acquiring the first information may include the step of acquiring a third image through the sensor, the step of identifying the location of the user in the third image, the step of identifying a rotational angle range of the sensor based on the location of the user and the field of view of the sensor, and the step of acquiring the first information through the sensor while the sensor rotates within the rotational angle range.

[0023] A non-transitory computer-readable medium storing instructions that cause a robot to perform an operation when executed by one or more processors of a robot including a projector according to an embodiment of the present disclosure, wherein the robot controls the projector to identify a plurality of candidate projection areas based on first information acquired by sensing the user's surroundings through the sensor, identify priorities of the plurality of candidate projection areas, project a first image on an area including the plurality of candidate projection areas based on a plurality of locations of the plurality of candidate projection areas and the priorities, and display second information about the priorities on the plurality of candidate projection areas, identify a candidate projection area selected from among the plurality of candidate projection areas based on a user input as a projection area, and project image content on the projection area through the projector.

[0024] The above and other aspects, features and advantages of the characteristic embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0025] FIG. 1A and FIG. 1B are block diagrams showing the configuration of a robot according to one or more embodiments.

[0026] FIG. 2 is a flowchart illustrating a method for projecting an image of a robot according to one or more embodiments.

[0027] FIG. 3 is a diagram illustrating an example of a bounding box for a user detected in an image according to one or more embodiments.

[0028] FIG. 4 is a diagram illustrating an example of a method for a robot to sense an area around a user according to one or more embodiments.

[0029] FIGS. 5A and 5B are drawings illustrating an example of an operation of a robot identifying multiple candidate projection areas according to one or more embodiments.

[0030] FIGS. 6A, 6B, 6C, and 6D are drawings illustrating an example of a method for a robot to position a candidate projection area within a pixel range of an image according to one or more embodiments.

[0031] FIG. 7 is a drawing illustrating an example of a method for a robot to generate an image to be projected using a projector according to one or more embodiments.

[0032] FIG. 8 is a drawing illustrating an example of an operation of a robot projecting an image according to one or more embodiments.

[0033] FIG. 9 is a drawing illustrating an example of a method for generating multiple images to be projected by a robot using a projector according to one or more embodiments.

[0034] FIGS. 10A, 10B, 10C and 10D are drawings illustrating an example of an operation of a robot projecting multiple images according to one or more embodiments.

[0035] FIG. 11 is a drawing illustrating an example of an operation of a robot projecting image content into a projection area according to one or more embodiments.

[0036] FIGS. 12a, 12b, 12c and 12d are drawings illustrating an example of an operation of a robot projecting multiple images according to one or more embodiments.

[0037] FIG. 13 is a block diagram illustrating a configuration of a robot according to one or more embodiments.

[0038] FIG. 14 is a flowchart illustrating a method for projecting an image of a robot according to one or more embodiments.

[0039] The embodiments described in this specification and the configurations illustrated in the drawings are examples of embodiments, and various modifications may be made without departing from the scope and spirit of this specification.

[0040] Hereinafter, terms used in this specification will be briefly described, and the present disclosure will be described in detail. In this disclosure, the expression “at least one of a, b, or c” can refer to “a,” “b,” “c,” “a and b,” “a and c,” “b and c,” “all of a, b, and c,” or variations thereof.

[0041] The terms used in this disclosure have been selected from widely used terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in some cases, terms may have been arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should be defined based on their meaning and the overall content of the disclosure, rather than their names.

[0042] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein. Furthermore, terms containing ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but such components should not be limited by such terms. Such terms are used solely to distinguish one component from another.

[0043] When a part throughout the specification is said to "include" another part, this does not mean excluding other components, but rather that they may include other components, unless otherwise specifically stated. Furthermore, terms such as "part" and "module" used in the specification mean a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0044] The term "and / or" includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0045] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0046] In the present disclosure, the process of obtaining text based on a voice signal corresponding to a user's voice and obtaining information about the user's intent based on the text can be performed by an artificial intelligence model. The artificial intelligence model can be implemented in the form of an on-device embedded in a robot. However, the present invention is not limited thereto, and the artificial intelligence model can also be stored on a server connected to the robot. If the artificial intelligence model is stored on the server, the robot can transmit a voice signal corresponding to the user's voice to the server and receive information about the user's intent or a control command based on the user's intent from the server.

[0047] The present disclosure relates to a robot and an image projection method of the robot that determines a projection area around a user by considering the characteristics of the area around the user, and projects an image onto the projection area to provide various image contents to the user.

[0048] Below, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, similar reference numerals may be used to indicate similar elements.

[0049] The present disclosure will be described below with reference to the attached drawings.

[0050] FIG. 1A and FIG. 1B are block diagrams showing the configuration of a robot according to one or more embodiments.

[0051] Referring to FIG. 1A, the robot (100) may include a projector (110), a sensor (120), a driving unit (130), and a main module (140). Meanwhile, the configuration of the robot (100) illustrated in FIG. 1A is an example, and it is understood that some configurations may be added depending on the embodiment.

[0052] A robot (100) according to one or more embodiments may be a mobile robot (e.g., a mobile robot). The robot (100) may also be referred to as an autonomous driving device, a mobile device, etc., but is described as a robot (100) in the present disclosure. The robot (100) moving may include exploring its surroundings to detect its location and obstacles, and autonomously navigating within a space using the detected information. The space in which the robot (100) moves may include various indoor spaces in which the robot (100) can navigate, such as a home, an office, a hotel, a factory, a store, a supermarket, a restaurant, etc.

[0053] The robot (100) according to one or more embodiments may be implemented as various types of robots. For example, the robot (100) may be implemented as a robot vacuum cleaner that moves within a space and performs cleaning, a guide robot that guides a user along a path within a space or provides various information related to services provided within the space, a delivery robot or serving robot that transports loaded products to a location within a space, or a mobile projection device that can project images while moving within a location.

[0054] The projector (110) can project images. The images can include still images and dynamic images (e.g., video). Dynamic images can include various visual information indicating the movement of an object using a plurality of consecutive still images. In this case, each of the plurality of still images included in the video can mean a frame (or image frame).

[0055] The projector (110) can project an image onto a projection surface using light emitted from a light source. For example, the projector (110) can project an image using a CRT (Cathode-Ray Tube) method, an LCD (Liquid Crystal Display) method, a DLP (Digital Light Processing) method, or an LCoS (Liquid Crystal on Silicon) method. The projection surface may be a separately provided screen, but is not limited thereto, and may be various wall surfaces within the space where the robot (100) moves, one side of an object, etc.

[0056] The sensor (120) is configured to sense various information. One or more processors (143) can acquire various information based on the sensing values ​​of the sensor (120). For example, the information acquired by the sensor (120) may include image and depth information. The image may include RGB values ​​of each of a plurality of pixels included in the image. The depth information may include a depth map including depth values ​​of each of a plurality of pixels.

[0057] According to one or more embodiments, the sensor (120) may include a stereo camera. The sensor (120) may include an RGB-D camera. The sensor (120) may include an RGB camera and a Light Detection And Ranging (LiDAR) sensor. However, the present invention is not limited to this example, and the sensor (120) may include various sensors capable of acquiring images and depth information.

[0058] The driving unit (130) can control the movement of the robot (100). For example, the driving unit (130) can move the robot (100), stop the moving robot (100), and control the moving speed and / or moving direction of the robot (100).

[0059] For example, the driving type of the robot (100) may be a wheel type or a walking type.

[0060] The wheel type refers to the way the robot (100) moves by rotating the wheels. If the robot (100) is a wheel type robot, the robot (100) may include one or more wheels. The drive unit (130) may include a device that generates power to rotate the wheels. For example, the drive unit (130) may be implemented as a gasoline engine, a diesel engine, an LPG (liquefied petroleum gas) engine, or an electric motor, depending on the fuel (or energy source) used.

[0061] The walking type refers to the way the robot (100) moves through the movement of its legs. If the robot (100) is a walking type (e.g., a bipedal walking robot, a triped walking robot, a quadruped walking robot, etc.), the robot (100) may include two or more legs that support the robot (100). The legs may include a plurality of links and joints connected to the links. The driving unit (130) may include a device that generates power to raise or lower the legs by rotating the links around the joints. For example, the driving unit (130) may be implemented with a motor and / or an actuator.

[0062] Additionally, the driving unit (130) can control the movement of a part of the robot (100). The driving unit (130) can be coupled between a first part (e.g., body) and a second part (e.g., head, arm, etc.) of the robot (100). The driving unit (130) can rotate the second part. For example, the driving unit (130) can be implemented with a motor and / or an actuator.

[0063] The main module (140) is implemented in hardware and may include a communication interface (141), memory (142), one or more processors (143), and a control unit (144).

[0064] The communication interface (141) can perform data communication with electronic devices under the control of one or more processors (143). For example, the communication interface (141) can include a communication circuit that can perform data communication between the robot (100) and the electronic devices using at least one of data communication methods including wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliances (WiGig), and RF communication.

[0065] Memory (142) may store instructions, data structures, and program codes that can be read by one or more processors (143). Operations performed by one or more processors (143) may be implemented by executing instructions or codes of a program stored in memory (142).

[0066] The memory (142) may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), and may include a non-volatile memory including at least one of a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk, and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0067] The memory (142) may store one or more instructions and / or programs that cause the robot (100) to determine a projection area around the user and operate to project an image onto the projection area. For example, referring to FIG. 1B, the memory (142) may store instructions and / or programs for implementing the functions of the candidate projection area identification module (21), the priority determination module (22), and the projection module (23). The modules (21, 22, 23) illustrated in FIG. 1B may be configurations implemented by one or more processors (143) executing programs or commands stored in the memory (142). Therefore, the operations described below as being performed by the modules (151, 152, 153) may actually be performed by one or more processors (143).

[0068] One or more processors (143) can control the overall operations of the robot (100). For example, the one or more processors (143) can control the overall operations of the robot (100) to determine a projection area around the user and project an image onto the projection area by executing one or more instructions of a program stored in the memory (142). For example, the one or more processors (143) can perform various operations for the robot (100) to determine a projection area around the user and project an image onto the projection area, and transmit a signal regarding the operation result to the control unit (144).

[0069] The control unit (144) can control components of the robot (100). The control unit (144) can control components of the robot (100) (e.g., a projector (110), a sensor (120), and a driving unit (130)) based on signals provided from one or more processors (143). For example, the control unit (144) can generate a control signal using signals provided from one or more processors (143) and provide the control signal to components of the robot (100). Accordingly, the components of the robot (100) can perform operations corresponding to the operation results of one or more processors (143). The control unit (144) can be implemented with one or more ICs (e.g., a controller IC).

[0070] The one or more processors (143) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (143) may control one or any combination of other components of the robot (100) and may perform operations or data processing related to communication. The one or more processors (143) may execute one or more programs or instructions stored in the memory (142). For example, the one or more processors (143) may perform a method according to one or more embodiments by executing one or more instructions stored in the memory (142).

[0071] When a method according to one or more embodiments includes multiple operations, the multiple operations may be performed by one processor or multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a CPU) and the third operation may be performed by the second processor (e.g., an artificial intelligence-dedicated processor).

[0072] One or more processors (143) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors (143) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments.

[0073] When a method according to one or more embodiments includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in the multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0074] In one or more embodiments, a processor may mean a system on a chip (SoC) having one or more processors and other electronic components integrated therein, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein a core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but one or more embodiments are not limited thereto.

[0075] For convenience of explanation, one or more processors (143) are referred to as processors (143) below.

[0076] The processor (143) can identify a plurality of candidate projection areas around the user by executing the candidate projection area identification module (21). The candidate projection area may refer to an area for projecting an image around the user. The candidate projection area identification module (21) can identify a flat surface having a sufficient area, shape, color (e.g., saturation), etc. for projecting an image of a certain size among various surfaces of objects such as walls, home appliances, and furniture within the space where the robot (100) is located, as a candidate projection area. For example, the candidate projection area identification module (21) can identify a flat surface around the user and identify a plurality of candidate projection areas around the user by considering the aspect ratio of the projection image, the size and color of the area within the flat surface.

[0077] A more detailed description of the candidate projection area identification module (21) will be further described in the description of S210, S220, S230, S240, and S250 of FIG. 2.

[0078] The processor (143) can execute the priority determination module (22) to identify priorities of multiple candidate projection areas and provide information about the priorities of the multiple candidate projection areas to the user.

[0079] Priority can be information that determines the order in which a projection area can be selected from among multiple candidate projection areas. For example, priority can indicate the order in which a projected image is visible to the user when projected onto a candidate projection area.

[0080] The priority determination module (22) can determine the priorities of multiple candidate projection areas based on the characteristics of the multiple candidate projection areas. At this time, the characteristics of the candidate projection areas may include the size of the candidate projection areas and the distance between the user and the candidate projection areas.

[0081] In addition, the priority decision module (22) can project an image indicating the priority of each candidate projection area onto each of a plurality of candidate projection areas using a projector (110).

[0082] The image may include indicators corresponding to the priorities of the candidate projection areas.

[0083] An indicator may include numbers or letters indicating the priority of a candidate projection area. The letters may include letters representing numbers in a specific language or letters in a sequence (e.g., alphabetic characters). The indicator may include various graphical elements that can visually provide users with information about the priority of a candidate projection area using numbers or letters, and may be replaced with a graphical user interface (GUI) or an icon.

[0084] A more detailed description of the priority decision module (22) will be further described in the description of S260 and S270 of FIG. 2.

[0085] The processor (143) can execute the projection module (23) to project an image onto a projection area. The projection area may refer to an area where an image provided to the user is projected. A projection area may be determined from among a plurality of candidate projection areas based on user input. The projection module (23) can project an image onto the projection area using a projector (110).

[0086] A more detailed description of the projection module (23) will be further described in the description of S290 in FIG. 2.

[0087] FIG. 2 is a flowchart illustrating a method for projecting an image of a robot according to one or more embodiments.

[0088] Referring to FIG. 2, the processor (143) can sense the user's surroundings using a sensor (120) to obtain information about the user's surroundings (S210).

[0089] The processor (143) can acquire information by sensing the user's surroundings using the sensor (120) while the user remains within the field of view (FOV) of the sensor (120).

[0090] For example, the projection area may be preset to be determined in a space within the field of view of the robot (100) (e.g., the field of view of the sensor (120)). In this case, in order for the image content projected onto the projection area by the robot (100) to be visible to the user, the projection area may be determined within the space based on the user's position. Accordingly, the robot (100) may sense the user's surroundings using the sensor (120) while maintaining the user within the field of view of the sensor (120) in order to obtain information about the space based on the user's position.

[0091] The processor (143) can identify the rotational angle range of the sensor (120) for sensing the user's surroundings while the user remains within the field of view of the sensor (120). In addition, the processor (143) can obtain information using the sensor (120) while the sensor (120) rotates within the rotational angle range.

[0092] For example, the processor (143) can acquire an image using the sensor (120) and identify the rotational angle range of the sensor (120) based on the location of the user identified in the image and the field of view of the sensor (120).

[0093] The location of the user in the image may include the location of a bounding box for the user. The processor (143) may detect the user in the image and obtain a bounding box for the user.

[0094] A bounding box for a user may mean a rectangular box that includes a user detected in an image. For example, as shown in FIG. 3, a processor (143) may obtain an image (310) including a user (10) using a sensor (120). At this time, assuming that the coordinates of the upper left pixel of the image (310) are (0, 0), the position of the bounding box (320) may be expressed by the x, y coordinate values ​​of the pixel (321) corresponding to the upper left vertex of the bounding box (320) and the x, y coordinate values ​​of the pixel (322) corresponding to the lower right vertex of the bounding box (320).

[0095] Additionally, the processor (143) can identify the pixel distance between the center pixel of the image and the pixel of the bounding box.

[0096] The center pixel of an image may include the pixel located in the center of the image. For example, the coordinates of the upper left pixel of the image (310) acquired by the sensor (120) are (0,0), and the coordinates of the lower right pixel of the image are (x width , yheight ) is assumed to be x width , y height can be determined based on the resolution of the image (310) acquired by the sensor (120). At this time, the coordinates of the center pixel of the image are (x width / 2, y height / 2) is the same.

[0097] Pixel distance can include the Euclidean distance between two pixels. For example, the pixel distance between a pixel with coordinates (x1, y1) and a pixel with coordinates (x2, y2) is It can be expressed as follows.

[0098] At this time, the processor (143) can identify a pixel corresponding to each of the four sides of the bounding box and identify the pixel distance between the center pixel of the image and the identified pixel.

[0099] For example, the processor (143) can identify a pixel corresponding to the left side of the bounding box. The left side of the bounding box is a line connecting the pixel corresponding to the upper left vertex of the bounding box and the pixel corresponding to the lower left vertex of the bounding box.

[0100] The processor (143) can identify a pixel, among a plurality of pixels of the image, whose x-coordinate value is equal to the x-coordinate value of the pixel corresponding to the upper left corner of the bounding box and whose y-coordinate value is equal to the y-coordinate value of the center pixel of the image, as a pixel corresponding to the left side of the bounding box. In addition, the processor (143) can identify a pixel distance between the pixel corresponding to the left side of the bounding box and the center coordinates of the image.

[0101] For example, the coordinates of the pixel corresponding to the upper left corner of the bounding box are (xupper_left, yupper_left), and the coordinates of the center pixel of the image are (x c , y c), the processor (143) selects a plurality of pixels of the image whose coordinates are (xupper_left, y c ) can be identified as a pixel corresponding to the left side of the bounding box. Then, the processor (143) (xupper_left, y c ) and (x c , y c ) can identify the pixel distance between them.

[0102] Additionally, the processor (143) can identify a pixel corresponding to the right side of the bounding box. The right side of the bounding box is a line connecting the pixel corresponding to the upper right vertex of the bounding box and the pixel corresponding to the lower right vertex of the bounding box.

[0103] The processor (143) can identify a pixel, among a plurality of pixels of the image, whose x-coordinate value is equal to the x-coordinate value of the pixel corresponding to the lower right corner of the bounding box and whose y-coordinate value is equal to the y-coordinate value of the center pixel of the image, as a pixel corresponding to the right side of the bounding box. In addition, the processor (143) can identify a pixel distance between the pixel corresponding to the right side of the bounding box and the center coordinates of the image.

[0104] For example, the coordinates of the pixel corresponding to the lower right corner of the bounding box are (xlower_right, ylower_right), and the coordinates of the center pixel of the image are (x c , y c ), the processor (143) selects one of the plurality of pixels of the image whose coordinates are (xlower_right, y c ) can be identified as a pixel corresponding to the right side of the bounding box. Then, the processor (143) (xlower_right, y c ) and (x c , y c ) can identify the pixel distance between them.

[0105] Additionally, the processor (143) can identify a pixel corresponding to the upper side of the bounding box. The upper side of the bounding box is a line connecting the pixel corresponding to the upper left vertex of the bounding box and the pixel corresponding to the upper right vertex of the bounding box.

[0106] The processor (143) can identify a pixel, among a plurality of pixels of the image, whose x-coordinate value is equal to the x-coordinate value of the center pixel of the image and whose y-coordinate value is equal to the y-coordinate value of the pixel corresponding to the upper left corner of the bounding box, as a pixel corresponding to the upper side of the bounding box. In addition, the processor (143) can identify a pixel distance between the pixel corresponding to the right side of the bounding box and the center coordinate of the image.

[0107] For example, the coordinates of the pixel corresponding to the upper left corner of the bounding box are (xupper_left, yupper_left), and the coordinates of the center pixel of the image are (x c , y c ), the processor (143) determines whether the coordinates of the plurality of pixels of the image are (x c , yupper_left) can be identified as a pixel corresponding to the upper side of the bounding box. Then, the processor (143) (x c , yupper_left) and (x c , y c ) can identify the pixel distance between them.

[0108] Additionally, the processor (143) can identify a pixel corresponding to the lower side of the bounding box. The lower side of the bounding box is a line connecting the pixel corresponding to the lower left vertex of the bounding box and the pixel corresponding to the lower right vertex of the bounding box.

[0109] The processor (143) can identify a pixel, among a plurality of pixels of the image, whose x-coordinate value is equal to the x-coordinate value of the center pixel of the image and whose y-coordinate value is equal to the y-coordinate value of the pixel corresponding to the lower right corner of the bounding box, as a pixel corresponding to the lower side of the bounding box. In addition, the processor (143) can identify a pixel distance between the pixel corresponding to the lower side of the bounding box and the center coordinates of the image.

[0110] For example, the coordinates of the pixel corresponding to the lower right corner of the bounding box are (xlower_right, ylower_right), and the coordinates of the center pixel of the image are (x c , y c ), the processor (143) determines whether the coordinates of the plurality of pixels of the image are (x c , ylower_right) can be identified as a pixel corresponding to the lower side of the bounding box. Then, the processor (143) (x c , ylower_right) and (x c , y c ) can identify the pixel distance between them.

[0111] Additionally, the processor (143) can identify the rotational angle range of the sensor (120) based on the identified pixel distance and the focal length of the sensor (120).

[0112] Focal length may include the distance from the principal point of the lens to the image sensor.

[0113] The rotation angle range of the sensor (120) may include a horizontal rotation angle range and a vertical rotation angle range.

[0114] For example, the processor (143) can identify a horizontal rotation angle range based on the first pixel distance, the second pixel distance, and the focal distance.

[0115] At this time, the first pixel distance is the pixel distance between the pixel corresponding to the left side of the bounding box and the center coordinate of the image, and the second pixel distance is the pixel distance between the pixel corresponding to the right side of the bounding box and the center coordinate of the image.

[0116] And, the horizontal rotation angle range may include an angular range between a maximum rotation angle of the sensor (120) in the right direction and a maximum rotation angle of the sensor (120) in the left direction. For example, the horizontal rotation angle range may include an angular range between a maximum rotation angle of the sensor (120) in the right direction and a maximum rotation angle of the sensor (120) in the left direction based on a shooting angle of the sensor (120) that captured an image from which a bounding box was acquired.

[0117] At this time, the maximum rotation angle of the sensor (120) in the right direction may include a rotation angle of the sensor (120) such that when an image is acquired by rotating the sensor (120) in the right direction, the left side of the bounding box for the user is located at the left edge of the acquired image. In addition, the maximum rotation angle of the sensor (120) in the left direction is a rotation angle of the sensor (120) such that when an image is acquired by rotating the sensor (120) in the left direction, the right side of the bounding box for the user is located at the right edge of the acquired image.

[0118] For example, if the x-coordinate value of the pixel corresponding to the left side of the bounding box is greater than the x-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the right direction is (horizontal FOV of the sensor) / 2 + tan -1 (first pixel distance / focal length). In addition, if the x-coordinate value of the pixel corresponding to the left side of the bounding box is smaller than the x-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the right direction is (horizontal FOV of the sensor) / 2 - tan -1(first pixel distance / focal length). In addition, the processor (143) can identify that the maximum rotation angle of the sensor (120) in the right direction is (horizontal FOV of the sensor) / 2 when the x-coordinate value of the pixel corresponding to the left side of the bounding box is equal to the x-coordinate value of the center pixel of the image.

[0119] And, if the x-coordinate value of the pixel corresponding to the right side of the bounding box is greater than the x-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the left direction is (horizontal FOV of the sensor) / 2 - tan -1 (second pixel distance / focal length). In addition, if the x-coordinate value of the pixel corresponding to the right side of the bounding box is smaller than the x-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the left direction is (horizontal FOV of the sensor) / 2 + tan -1 (second pixel distance / focal length). In addition, the processor (143) can identify that the maximum rotation angle of the sensor (120) in the left direction is (horizontal FOV of the sensor) / 2 when the x-coordinate value of the pixel corresponding to the right side of the bounding box is the same as the x-coordinate value of the center pixel of the image.

[0120] For example, the processor (143) can identify the vertical rotation angle range based on the third pixel distance, the fourth pixel distance, and the focal distance.

[0121] At this time, the third pixel distance is the pixel distance between the pixel corresponding to the upper side of the bounding box and the center coordinate of the image, and the fourth pixel distance is the pixel distance between the pixel corresponding to the lower side of the bounding box and the center coordinate of the image.

[0122] And, the vertical rotation angle range may include an angular range between a maximum rotation angle of the sensor (120) in a downward direction and a maximum rotation angle of the sensor (120) in an upward direction. For example, the vertical rotation angle range may include an angular range between a maximum rotation angle of the sensor (120) in a downward direction and a maximum rotation angle of the sensor (120) in an upward direction based on a shooting angle of the sensor (120) that captured an image from which a bounding box is acquired.

[0123] At this time, the maximum rotation angle of the sensor (120) in the downward direction may include a rotation angle of the sensor (120) such that when an image is acquired by rotating the sensor (120) in the downward direction, the upper side of the bounding box for the user is located at the upper edge of the acquired image. In addition, the maximum rotation angle of the sensor (120) in the upward direction may include a rotation angle of the sensor (120) such that when an image is acquired by rotating the sensor (120) in the upward direction, the lower side of the bounding box for the user is located at the lower edge of the acquired image.

[0124] For example, if the y-coordinate value of the pixel corresponding to the upper side of the bounding box is less than the y-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the downward direction is (vertical FOV of the sensor) / 2 - tan -1 (3rd pixel distance / focal length). In addition, if the y-coordinate value of the pixel corresponding to the upper side of the bounding box is greater than the y-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the downward direction is (vertical FOV of the sensor) / 2 + tan -1(3rd pixel distance / focal length). In addition, the processor (143) can identify that the maximum rotation angle of the sensor (120) in the downward direction is (vertical FOV of the sensor) / 2 when the y-coordinate value of the pixel corresponding to the upper side of the bounding box is the same as the y-coordinate value of the center pixel of the image.

[0125] And, if the y-coordinate value of the pixel corresponding to the lower side of the bounding box is smaller than the y-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the upper direction is (vertical FOV of the sensor) / 2 + tan -1 (4th pixel distance / focal length). In addition, if the y-coordinate value of the pixel corresponding to the lower side of the bounding box is greater than the y-coordinate value of the center pixel of the image, the processor (143) determines that the maximum rotation angle of the sensor (120) in the upward direction is (vertical FOV of the sensor) / 2 - tan -1 (4th pixel distance / focal length). In addition, the processor (143) can identify that the maximum rotation angle of the sensor (120) in the upward direction is (vertical FOV of the sensor) / 2 when the y-coordinate value of the pixel corresponding to the lower side of the bounding box is the same as the y-coordinate value of the center pixel of the image.

[0126] And, the processor (143) can rotate the sensor (120) within a rotation angle range.

[0127] At this time, the rotation of the sensor (120) may include the sensor (120) being rotated and the sensor (120) being rotated as a part of the robot (100) on which the sensor (120) is placed is rotated.

[0128] For example, if the sensor (120) is disposed on the body of the robot (100), the processor (143) can control the driving unit (130) to rotate the body of the robot (100) within a rotation angle range. At this time, the sensor (120) can be rotated together with the body within the rotation angle range. In addition, if the robot (100) is composed of a first part (e.g., body) and a second part (e.g., head, arm, etc.) and the sensor (120) is disposed on the second part, the processor (143) can control the driving unit (130) to rotate the second part of the robot (100) within a rotation angle range. At this time, the sensor (120) can be rotated together with the second part within the rotation angle range.

[0129] For example, the processor (143) can rotate the sensor (120) in the right and left directions within a horizontal rotation angle range, and can rotate the sensor (120) in the downward and upward directions within a vertical rotation angle range.

[0130] In addition, the processor (143) can obtain information using the sensor (120) while the sensor (120) rotates. At this time, the information can include image and depth information.

[0131] For example, referring to FIG. 4, the sensor (120) can sense an area (410) around the user (10) while the sensor (120) rotates.

[0132] At this time, the region (411) is a region sensed by the sensor (120) rotated based on the maximum rotation angle in the left direction and the maximum rotation angle in the upward direction. In addition, the region (412) is a region sensed by the sensor (120) rotated based on the maximum rotation angle in the right direction and the maximum rotation angle in the upward direction. In addition, the region (413) is a region sensed by the sensor (120) rotated based on the maximum rotation angle in the left direction and the maximum rotation angle in the downward direction. In addition, the region (414) is a region sensed by the sensor (120) rotated based on the maximum rotation angle in the right direction and the maximum rotation angle in the downward direction.

[0133] The processor (143) can sense the user's surroundings using a sensor (120) and generate a three-dimensional map of the user's surroundings based on the acquired information (S220).

[0134] For example, the processor (143) may obtain a three-dimensional map composed of a point cloud based on images and depth information obtained using the sensor (120). According to one or more embodiments, the processor (143) may obtain the three-dimensional map using 3D reconstruction. The point cloud may include a set of points in three-dimensional space. Each point in the point cloud may include x, y, z coordinate values ​​and RGB values ​​of the point.

[0135] The processor (143) can identify a plane in an area around the user based on a three-dimensional map (S230).

[0136] At this time, the processor (143) can identify a plane around the user using RANSAC (RANdom SAmple Consensus).

[0137] For example, the processor (143) can randomly select three points among the points included in the area of ​​the 3D map. Then, the processor (143) can identify a plane defined by the selected points. The plane can be expressed by a plane equation such as ax+by+cz+d=0. Then, the processor (143) can identify the distance between the remaining points except for the three points among the points included in the area and the plane, and can identify an inlier based on the identified distance. The inlier can include a point among the points included in the area whose distance from the plane is less than or equal to a threshold value.

[0138] The processor (143) can identify a plurality of planes and the number of inliers in each of the plurality of planes by performing the above-described process multiple times. In addition, the processor (143) can identify a plane with the largest number of inliers among the plurality of planes based on the number of inliers in the plurality of planes.

[0139] In addition, the processor (143) can identify a ratio of inliers of the identified plane to the points included in the area based on the number of points included in the area and the number of inliers of the identified plane (e.g., the plane with the largest number of inliers). At this time, the processor (143) can identify the area as being planar if the identified ratio is greater than or equal to a threshold value, and can identify the area as not being planar if the identified ratio is less than the threshold value.

[0140] Through this process, the processor (143) can identify a planar area in a three-dimensional map of the user's surroundings.

[0141] The processor (143) can identify a plurality of areas on the identified plane whose aspect ratio is equal to the aspect ratio of the projection image (S240).

[0142] At this time, the aspect ratio of the projected image (or projected image) may include the aspect ratio of the image projected from the projector (110). For example, the aspect ratio may include various ratios such as 4:3, 5:4, or 16:9.

[0143] For example, the processor (143) can identify points included in a plane based on a 3D map, and randomly select a point from among the points included in the plane. Then, the processor (143) can expand an area from the selected point so that the selected point becomes the center, and generate a candidate area on the plane. At this time, the area is expanded within the same plane, and the aspect ratio of the expanded area may be the same as the aspect ratio of the projection image. The processor (143) can generate a plurality of candidate areas on the plane by performing this process multiple times. The candidate area may mean a candidate area on which an image can be projected, in that it is a rectangular area on the plane that has the same aspect ratio as the projection image.

[0144] And, the processor (143) can identify multiple regions among multiple candidate regions based on the location and size of the multiple candidate regions.

[0145] For example, the processor (143) may determine a plurality of regions by selecting a plurality of regions in descending order of size among the plurality of candidate regions based on the sizes of the plurality of candidate regions. At this time, the number of regions to be selected may be preset.

[0146] At this time, the processor (143) may not select a candidate region that is included within another candidate region on the plane based on the locations of the multiple candidate regions. Including one region in another region may include the entire region being located within the other region. Accordingly, if at least a portion of one region is located outside another region, the region may be considered not to be included in the other region.

[0147] The processor (143) can identify multiple candidate projection areas among multiple areas (S250).

[0148] For example, the processor (143) may identify a plurality of candidate projection areas among a plurality of areas based on characteristics of the plurality of areas. At this time, the characteristics of the areas may include saturation of the areas. When an area having saturation greater than a threshold value is identified among the plurality of areas based on RGB values ​​of points included in the plurality of areas, the processor (143) may identify the remaining areas excluding the identified area among the plurality of areas as being the plurality of candidate projection areas.

[0149] For example, the processor (143) may obtain RGB values ​​of points included in a plurality of regions based on a three-dimensional map, and identify the saturation of the plurality of regions based on the RGB values. For example, the processor (143) may identify the saturation of each point based on the RGB values ​​of the points included in each region, and may identify the saturation of each region by calculating an average value of the identified saturation. At this time, the processor (143) may identify the saturation of the region using the RGB values ​​of all points included in the region, or may select a sample point among all points included in the region, and identify the saturation of the region using the RGB values ​​of the selected sample point.

[0150] In addition, the processor (143) can compare the saturation of each of the plurality of regions with a threshold value to identify whether there is a region among the plurality of regions whose saturation is higher than the threshold value. In addition, when a region among the plurality of regions whose saturation is higher than the threshold value is identified, the processor (143) can determine the remaining regions, excluding the identified region, among the plurality of regions as the plurality of candidate projection regions.

[0151] Since saturation indicates the intensity of a color, a high saturation in an area can indicate a deep color. If an image is projected onto a dark area, the projected image may not be visible to the user, so areas with high saturation may be excluded from the candidate projection area.

[0152] For example, referring to FIG. 5a, the processor (143) can identify a plurality of areas (511, 512, 513, 514, 515, 516, 517, 518) around the user (10). At this time, each of the plurality of areas is located on a plane, and an aspect ratio of each area may be the same as an aspect ratio of the projection image. In addition, referring to FIG. 5b, the processor (143) can identify a plurality of candidate projection areas (512, 513, 515, 517) among the plurality of areas (511, 512, 513, 514, 515, 516, 517, 518). At this time, the saturation of the candidate projection areas may be less than a threshold value.

[0153] The processor (143) identifies the priorities of multiple candidate projection areas (S260).

[0154] Priority may be information that determines the order in which a projection area can be selected from among multiple candidate projection areas. For example, priority may include the ranking of the candidate projection areas in which the projected image can be viewed by the user when the image is projected onto the candidate projection area.

[0155] For example, the processor (143) may identify priorities of multiple candidate projection areas based on characteristics of the multiple candidate projection areas. In this case, the characteristics of the candidate projection areas may include the size of the candidate projection areas and the distance between the user and the candidate projection areas.

[0156] The processor (143) can identify the size of each of the plurality of candidate projection areas.

[0157] As described above, the candidate projection area is generated by extending from a point on the plane, and the point may be the center of the candidate projection area. The processor (143) can identify the size of the candidate projection area based on the extent to which the candidate projection area extends from the point.

[0158] Additionally, the processor (143) can identify the distance between the user and each of the plurality of candidate projection areas.

[0159] For example, the processor (143) can identify x, y, z coordinate values ​​corresponding to the user's location on a 3D map. As described above, the processor (143) can generate a 3D map of the user's surroundings using image and depth information. In the process of generating the 3D map, the processor (143) can obtain x, y, z coordinate values ​​on the 3D map corresponding to the user's x, y coordinate values, and identify the x, y, z coordinate values ​​corresponding to the user's location on the 3D map. At this time, the user's x, y coordinate values ​​may include coordinate values ​​corresponding to the user's location acquired from the image. For example, the user's x, y coordinate values ​​acquired from the image may be the x, y coordinate values ​​of the center pixel of a bounding box for the user. If the coordinates of the pixel corresponding to the upper left corner of the bounding box are (xupper_left, yupper_left) and the coordinates of the pixel corresponding to the lower right corner of the bounding box are (xlower_right, ylower_right), then the coordinates of the center pixel of the bounding box can be expressed as ((xlower_right-xupper_left) / 2, (ylower_right-yupper_left) / 2).

[0160] In addition, the processor (143) can identify the distance between the user and the candidate projection area by using the x, y, z coordinate values ​​of the point corresponding to the center of the candidate projection area and the x, y, z coordinate values ​​corresponding to the user's position. At this time, the distance may include the Euclidean distance between the two coordinate values.

[0161] The processor (143) can identify priorities of the plurality of candidate projection areas based on the size of each of the plurality of candidate areas and the distance between the user and each of the plurality of candidate projection areas.

[0162] For example, the processor (143) can identify the score of a candidate projection area based on the distance between the user and the candidate projection area and the size of the candidate projection area. At this time, the score can be calculated based on the following mathematical expression 1.

[0163]

[0164] Here, D is the distance between the user and the candidate projection area, S is the size of the candidate projection area, and w1 and w2 are weights.

[0165] In addition, the processor (143) can identify the priorities of the multiple candidate projection areas based on the scores of the multiple candidate projection areas, such that the candidate projection area with the highest score is given a higher priority. Accordingly, the closer the candidate projection area is to the user and the larger the area, the higher the priority it can have among the multiple candidate projection areas.

[0166] The processor (143) displays information about the priorities of the plurality of candidate projection areas in the plurality of candidate projection areas (S270).

[0167] For example, the processor (143) controls the projector (110) to project an image onto an area including a plurality of candidate projection areas based on the positions of the plurality of candidate projection areas and the priorities of the plurality of candidate projection areas, thereby simultaneously displaying information on the priorities of the plurality of candidate projection areas onto the plurality of candidate projection areas or sequentially displaying information on the priorities of the plurality of candidate projection areas onto the plurality of candidate projection areas.

[0168] To this end, the processor (143) can identify whether a plurality of candidate projection areas are within an area where an image is to be projected by the projector (110).

[0169] For example, the processor (143) can obtain 3D coordinate values ​​of points corresponding to vertices of each of a plurality of candidate projection areas based on a 3D map. At this time, the 3D coordinate values ​​are coordinate values ​​on the world coordinate system.

[0170] And, the processor (143) can convert the three-dimensional coordinate values ​​into two-dimensional coordinate values. At this time, the two-dimensional coordinate values ​​may be coordinate values ​​on a pixel coordinate system for an image to be projected by the projector (110). For example, the processor (143) can convert the three-dimensional coordinate values ​​into two-dimensional coordinate values ​​using camera calibration, as in the following mathematical expression 2.

[0171]

[0172] Here, X, Y, Z are 3D coordinates, x, y are 2D coordinates, K is an intrinsic matrix, [R│t] is an extrinsic matrix, and s is a scale factor.

[0173] The intrinsic matrix may include internal parameters of the camera, such as the focal length (e.g., fx, fy) of the camera (e.g., sensor (120)) and the principal point (e.g., cx, cy) of the camera.

[0174] The Extrinsic Matrix is ​​a matrix for transforming the world coordinate system into the camera coordinate system, and may include external parameters of the camera, such as the rotation and translation of the camera. In this case, the rotation and translation of the camera represent the pose of the camera. For example, the processor (143) may obtain parameters related to the rotation and translation of the camera using Visual Odometry to estimate the pose of the camera.

[0175] In addition, the processor (143) can identify whether the two-dimensional coordinate values ​​corresponding to the candidate projection area are within the pixel range of the image to be projected by the projector (110). At this time, the two-dimensional coordinate values ​​corresponding to the candidate projection area may include two-dimensional coordinate values ​​converted from the three-dimensional coordinate values ​​of the candidate projection area.

[0176] For example, the coordinates of the upper left pixel of the image to be projected by the projector (110) are (0,0), and the coordinates of the lower right pixel of the image are (x width , y height ) is assumed to be x width , y height can be determined based on the resolution of the image to be projected by the projector (110). In this case, the pixel range of the image is 0≤x≤x width , 0≤y≤y height . At this time, 0≤x≤x width is the pixel range of the x-coordinate value, 0≤y≤y height is the pixel range of the y-coordinate value.

[0177] The processor (143) can identify that the candidate projection area is within the area on which an image is to be projected by the projector (110) if, among the two-dimensional coordinate values ​​corresponding to the candidate projection area, the x-coordinate value is within the pixel range of the x-coordinate value and the y-coordinate value is within the pixel range of the y-coordinate value. In addition, the processor (143) can identify that, among the two-dimensional coordinate values ​​corresponding to the candidate projection area, the x-coordinate value is not within the pixel range of the x-coordinate value or the y-coordinate value is not within the pixel range of the y-coordinate value.

[0178] In this way, the processor (143) can identify whether a plurality of candidate projection areas are within the area where an image is to be projected by the projector (110).

[0179] If the processor (143) identifies that a candidate projection area among a plurality of candidate projection areas is not within an area where an image is to be projected by the projector (110), the processor (143) can identify a rotation angle of the projector (110) that allows the candidate projection area to be positioned within an area where an image is to be projected by the projector (110), and can rotate the projector (110) based on the identified rotation angle. By rotating the projector (110), the plurality of candidate projection areas can be positioned within an area where an image is to be projected by the projector (110).

[0180] For example, the processor (143) can identify the rotation angle of the projector (110) based on the x,y coordinate values ​​of a candidate projection area that is identified as not being within the area where the image is to be projected.

[0181] At this time, the rotation angle of the projector (110) may include a horizontal rotation angle and a vertical rotation angle.

[0182] According to one or more embodiments, the processor (143) may identify, among the x, y coordinate values ​​of the candidate projection area, a coordinate value that is outside the pixel range of the image. For example, the processor (143) may identify, among the x coordinate values ​​of the candidate projection area, an x ​​coordinate value that is outside the pixel range of the x coordinate value of the image, and may identify, among the y coordinate values ​​of the candidate projection area, a y coordinate value that is outside the pixel range of the y coordinate value of the image.

[0183] And, if at least one x-coordinate value outside the pixel range of the x-coordinate value of the image is identified, the processor (143) determines the lower limit (e.g., 0) or upper limit (e.g., x) of the pixel range of the x-coordinate value for each of the identified x-coordinate values. width ) can identify the distance between them. At this time, the processor (143) can identify the distance between the x-coordinate value and the value that is close to the x-coordinate value among the lower and upper limits.

[0184] Additionally, if at least one y-coordinate value outside the pixel range of the y-coordinate value of the image is identified, the processor (143) compares each of the identified y-coordinate values ​​with the lower limit (e.g., 0) or upper limit (e.g., y) of the pixel range of the y-coordinate value. height ) can identify the distance between them. At this time, the processor (143) can identify the distance between the y-coordinate value and the value that is close to the y-coordinate value among the lower and upper limits.

[0185] FIGS. 6a, 6b, 6c and 6d are drawings illustrating examples of a plurality of areas (620, 630, 640, 650) defined by two-dimensional coordinate values ​​corresponding to a plurality of candidate projection areas on a pixel coordinate system for an image (610) to be projected by a projector (110).

[0186] For example, region (620) is a region defined by two-dimensional coordinate values ​​corresponding to a first candidate projection region (e.g., 512 in FIG. 5b). Region (630) is a region defined by two-dimensional coordinate values ​​corresponding to a second candidate projection region (e.g., 513 in FIG. 5b). Region (640) is a region defined by two-dimensional coordinate values ​​corresponding to a third candidate projection region (e.g., 515 in FIG. 5b). Region (650) is a region defined by two-dimensional coordinate values ​​corresponding to a fourth candidate projection region (e.g., 517 in FIG. 5b).

[0187] Referring to FIG. 6A, the processor (143) can identify that the x-coordinate value of the pixel (621) corresponding to the upper left corner of the region (620) and the x-coordinate value of the pixel (622) corresponding to the lower left corner of the region (620) are outside the pixel range of the x-coordinate value of the image (610). At this time, the processor (143) can identify the distance between the x-coordinate value of the pixel (621) and the lower limit of the pixel range of the x-coordinate value of the image (610), and can identify the distance between the x-coordinate value of the pixel (622) and the lower limit of the pixel range of the x-coordinate value of the image (610).

[0188] And, the processor (143) can identify the rotation angle of the projector (110) based on a distance with a large size among the identified distances. At this time, the rotation angle of the projector (110) may include a rotation angle in the left direction. For example, information about the rotation angle of the projector (110) so that the x-coordinate value is positioned within the pixel range of the x-coordinate value of the image according to the distance between the x-coordinate value and the pixel range of the x-coordinate value may be stored in the memory (142). The processor (143) can identify the rotation angle of the projector (110) based on the information about the rotation angle of the projector (110) according to the identified distance.

[0189] In addition, referring to FIG. 6B, the processor (143) can identify that the y-coordinate value of the pixel (631) corresponding to the upper left corner of the region (630) and the y-coordinate value of the pixel (632) corresponding to the upper right corner of the region (630) are outside the pixel range of the y-coordinate value of the image (610). At this time, the processor (143) can identify the distance between the y-coordinate value of the pixel (631) and the lower limit of the pixel range of the y-coordinate value of the image (610), and can identify the distance between the y-coordinate value of the pixel (632) and the lower limit of the pixel range of the y-coordinate value of the image (610).

[0190] And, the processor (143) can identify the rotation angle of the projector (110) based on a distance with a large size among the identified distances. At this time, the rotation angle of the projector (110) may include a rotation angle in the upward direction. For example, information about the rotation angle of the projector (110) so that the y-coordinate value is positioned within the pixel range of the y-coordinate value of the image according to the distance between the y-coordinate value and the pixel range of the y-coordinate value may be stored in the memory (142). The processor (143) can identify the rotation angle of the projector (110) based on the information about the rotation angle of the projector (110) according to the identified distance.

[0191] In addition, referring to FIG. 6c, the processor (143) can identify that the y-coordinate value of the pixel (641) corresponding to the lower left corner of the region (640) and the y-coordinate value of the pixel (642) corresponding to the lower right corner of the region (640) are outside the pixel range of the y-coordinate value of the image (610). At this time, the processor (143) can identify the distance between the y-coordinate value of the pixel (641) and the upper limit of the pixel range of the y-coordinate value of the image (610), and can identify the distance between the y-coordinate value of the pixel (642) and the upper limit of the pixel range of the y-coordinate value of the image (610).

[0192] And, the processor (143) can identify the rotation angle of the projector (110) based on a distance with a large size among the identified distances. At this time, the rotation angle of the projector (110) may include a rotation angle in the downward direction. For example, information about the rotation angle of the projector (110) so that the y-coordinate value is positioned within the pixel range of the y-coordinate value of the image according to the distance between the y-coordinate value and the pixel range of the y-coordinate value may be stored in the memory (142). The processor (143) can identify the rotation angle of the projector (110) based on the information about the rotation angle of the projector (110) according to the identified distance.

[0193] In addition, referring to FIG. 6d, the processor (143) can identify that the x-coordinate value of the pixel (651) corresponding to the upper right corner of the region (650) and the x-coordinate value of the pixel (652) corresponding to the lower right corner of the region (650) are outside the pixel range of the x-coordinate value of the image (610). At this time, the processor (143) can identify the distance between the x-coordinate value of the pixel (651) and the upper limit of the pixel range of the x-coordinate value of the image (610), and can identify the distance between the x-coordinate value of the pixel (652) and the upper limit of the pixel range of the x-coordinate value of the image (610).

[0194] And, the processor (143) can identify the rotation angle of the projector (110) based on a distance with a large size among the identified distances. At this time, the rotation angle of the projector (110) may include a rotation angle in the right direction. For example, information about the rotation angle of the projector (110) so that the x-coordinate value is positioned within the pixel range of the x-coordinate value of the image according to the distance between the x-coordinate value and the pixel range of the x-coordinate value may be stored in the memory (142). The processor (143) can identify the rotation angle of the projector (110) based on the information about the rotation angle of the projector (110) according to the identified distance.

[0195] In this way, the processor (143) can identify a rotation angle of the projector (110) in the left, right, upward, or downward direction so that at least one candidate projection area located outside the pixel range of the image is located within the pixel range of the image. At this time, the processor (143) can identify the rotation angle of the projector (110) so that an area within the pixel range of the image does not move outside the pixel range of the image according to the rotation of the projector (110). Then, the processor (143) can rotate the projector (110) based on the identified rotation angle.

[0196] And, the processor (143) can generate an image to be projected by the projector (110) based on a plurality of sub-images.

[0197] According to one or more embodiments, the processor (143) can align a plurality of sub-images to a plurality of areas of an image to be projected by the projector (110), thereby generating an image in which the plurality of sub-images are included in a plurality of areas.

[0198] For example, the processor (143) may identify a plurality of regions corresponding to the plurality of candidate projection regions in an image to be projected onto an area including the plurality of candidate projection regions based on the positions of the plurality of candidate projection regions. In this case, the region corresponding to the candidate projection region may include a region defined by two-dimensional coordinate values ​​corresponding to the candidate projection region on the image to be projected.

[0199] And, the processor (143) can obtain multiple sub-images corresponding to multiple areas.

[0200] The colors of the plurality of sub-images may be different from the colors of the remaining areas of the image to be projected by the projector (110). For example, the colors of the plurality of sub-images may be a first color, and the colors of the remaining areas may be a second color different from the first color.

[0201] Each of the plurality of sub-images may include an indicator corresponding to the priority of each of the plurality of candidate projection areas. The indicator may include numbers or letters indicating the priority of the candidate projection areas. The letters may include letters that represent numbers in a language or letters that have a sequence (e.g., alphabets). The indicator may include various graphic elements that can visually provide information to the user about the priority of the candidate projection areas using numbers or letters, and may be replaced with expressions such as GUIs or icons. The color of the indicator may be different from the color of the background area. In this case, the background area may include the remaining area of ​​the sub-image excluding the indicator.

[0202] In addition, the processor (143) can align multiple sub-images to multiple areas. At this time, the processor (143) can obtain multiple transformation matrices for aligning multiple sub-images to multiple areas. The transformation matrix may include a homography matrix.

[0203] For example, the processor (143) can obtain a transformation matrix for aligning a sub-image to an area corresponding to the sub-image based on the following mathematical expression 3.

[0204]

[0205] Here, H is a Homography Matrix, (x, y) are the coordinates of pixels corresponding to the vertices of the sub-image, and (x', y') are the coordinates of the area corresponding to the sub-image in the image to be projected by the projector (110).

[0206] In addition, the processor (143) can apply a plurality of transformation matrices to a plurality of sub-images to generate an image in which the plurality of sub-images are included in a plurality of areas.

[0207] For example, referring to FIG. 7, the processor (143) may apply a transformation matrix (H1) to the RGB values ​​of pixels of the first sub-image (710) corresponding to the first candidate projection area, thereby aligning the first sub-image (710) with an area (751) corresponding to the first candidate projection area in an image (750) to be projected by the projector (110). The first candidate projection area is a candidate projection area with a priority of 1, and the first sub-image (710) may include a number 1 (711) indicating the priority of the first candidate projection area.

[0208] Additionally, the processor (143) may apply a transformation matrix (H2) to the RGB values ​​of pixels of the second sub-image (720) corresponding to the second candidate projection area, thereby aligning the second sub-image (720) with an area (752) corresponding to the second candidate projection area in an image (750) to be projected by the projector (110). The second candidate projection area is a candidate projection area with a priority of 2, and the second sub-image (720) may include a number 2 (721) indicating the priority of the second candidate projection area.

[0209] Additionally, the processor (143) may apply a transformation matrix (H3) to the RGB values ​​of pixels of the third sub-image (730) corresponding to the third candidate projection area, thereby aligning the third sub-image (730) with an area (753) corresponding to the third candidate projection area in an image (750) to be projected by the projector (110). The third candidate projection area is a candidate projection area with a priority of 3, and the third sub-image (730) may include a number 3 (731) indicating the priority of the third candidate projection area.

[0210] Additionally, the processor (143) may apply a transformation matrix (H4) to the RGB values ​​of pixels of the fourth sub-image (740) corresponding to the fourth candidate projection area, thereby aligning the fourth sub-image (740) with an area (754) corresponding to the fourth candidate projection area in an image (750) to be projected by the projector (110). The fourth candidate projection area is a candidate projection area with a priority of 4, and the fourth sub-image (740) may include a number 4 (741) indicating the priority of the fourth candidate projection area.

[0211] In addition, the processor (143) can control the projector (110) to project an image including multiple sub-images in multiple areas. At this time, when the image is projected, multiple sub-images included in multiple areas of the image can be projected onto each of multiple candidate projection areas.

[0212] For example, referring to FIG. 8, the processor (143) can control the projector (110) to project an image (750) including first to fourth sub-images (710, 720, 730, 740).

[0213] At this time, when the image (750) is projected, the first sub-image (710) may be positioned in the first candidate projection area (810), the second sub-image (720) may be positioned in the second candidate projection area (820), the third sub-image (730) may be positioned in the third candidate projection area (830), and the fourth sub-image (740) may be positioned in the fourth candidate projection area (840).

[0214] According to one or more implementation examples, the processor (143) can align sub-images corresponding to areas of an image to be projected by the projector (110) to generate multiple images including sub-images for each area.

[0215] For example, the processor (143) may identify an area corresponding to a candidate projection area in an image to be projected on the area based on the location of the candidate projection area for each candidate projection area. In this case, the area corresponding to the candidate projection area may include an area defined by two-dimensional coordinate values ​​corresponding to the candidate projection area on the image to be projected.

[0216] And, the processor (143) can obtain multiple sub-images corresponding to multiple areas.

[0217] The colors of the plurality of sub-images may be different from the colors of the remaining areas of the image to be projected by the projector (110). For example, the colors of the plurality of sub-images may be a first color, and the colors of the remaining areas may be a second color different from the first color.

[0218] Each of the plurality of sub-images may include an indicator corresponding to the priority of each of the plurality of candidate projection areas. The indicator may include numbers or letters indicating the priority of the candidate projection areas. The letters may include letters that represent numbers in a language or letters that have a sequence (e.g., alphabets). The indicator may include various graphic elements that can visually provide information to the user about the priority of the candidate projection areas using numbers or letters, and may be replaced with expressions such as GUIs or icons. The color of the indicator may be different from the color of the background area. In this case, the background area may include the remaining area of ​​the sub-image excluding the indicator.

[0219] And, the processor (143) can align the sub-image to the region. At this time, the processor (143) can obtain a transformation matrix for aligning the sub-image corresponding to the region to the region based on mathematical expression 3. And, the processor (143) can apply the transformation matrix to the sub-image to generate an image in which the sub-image is included in the region.

[0220] For example, referring to FIG. 9, the processor (143) may apply a transformation matrix (H1) to the RGB values ​​of pixels of the first sub-image (910) corresponding to the first candidate projection area, thereby aligning the first sub-image (910) with an area (951) corresponding to the first candidate projection area in an image (950) to be projected by the projector (110). The first candidate projection area is a candidate projection area with a priority of 1, and the first sub-image (910) may include a number 1 (911) indicating the priority of the first candidate projection area.

[0221] Additionally, the processor (143) may apply a transformation matrix (H2) to the RGB values ​​of pixels of the second sub-image (920) corresponding to the second candidate projection area, thereby aligning the second sub-image (920) with an area (961) corresponding to the second candidate projection area in an image (960) to be projected by the projector (110). The second candidate projection area is a candidate projection area with a priority of second, and the second sub-image (920) may include a number 2 (921) indicating the priority of the first candidate projection area.

[0222] Additionally, the processor (143) may apply a transformation matrix (H3) to the RGB values ​​of pixels of the third sub-image (930) corresponding to the third candidate projection area, thereby aligning the third sub-image (930) with an area (971) corresponding to the third candidate projection area in an image (970) to be projected by the projector (110). The third candidate projection area is a candidate projection area with a priority of 3, and the third sub-image (930) may include a number 3 (931) indicating the priority of the third candidate projection area.

[0223] Additionally, the processor (143) may apply a transformation matrix (H4) to the RGB values ​​of pixels of the fourth sub-image (940) corresponding to the fourth candidate projection area, thereby aligning the fourth sub-image (940) with an area (981) corresponding to the fourth candidate projection area in an image (980) to be projected by the projector (110). The fourth candidate projection area is a candidate projection area with a priority of 4, and the fourth sub-image (940) may include a number 4 (941) indicating the priority of the first candidate projection area.

[0224] In addition, the processor (143) can control the projector (110) so that each sub-image sequentially projects multiple images included in each area. At this time, when the image is projected, the sub-images included in the area of ​​the image can be projected on each candidate projection area. The processor (143) can control the projector (110) so that, based on the priorities of the multiple candidate projection areas, the images including the sub-images corresponding to the candidate projection areas with the highest priority are projected first.

[0225] For example, referring to FIG. 10A, the processor (143) may control the projector (110) to project an image (950) including a first sub-image (910). When the image (950) is projected, the first sub-image (910) may be located in a first candidate projection area (1010).

[0226] Next, referring to FIG. 10b, the processor (143) may control the projector (110) to project an image (960) including a second sub-image (920). At this time, when the image (960) is projected, the second sub-image (920) may be located in the second candidate projection area (1020).

[0227] Next, referring to FIG. 10c, the processor (143) may control the projector (110) to project an image (970) including a third sub-image (930). At this time, when the image (970) is projected, the third sub-image (930) may be located in the third candidate projection area (1030).

[0228] Next, referring to FIG. 10d, the processor (143) may control the projector (110) to project an image (980) including a fourth sub-image (940). At this time, when the image (980) is projected, the fourth sub-image (940) may be located in the fourth candidate projection area (1040).

[0229] The processor (143) identifies a candidate projection area selected from among multiple candidate projection areas based on user input as a projection area (S280).

[0230] User input can be entered in a variety of ways.

[0231] According to one or more embodiments, the processor (143) may receive a voice signal corresponding to a user's voice. For example, when the processor (143) receives a voice signal corresponding to a user's voice, the processor (143) may convert the voice signal into text and identify an indicator included in the user's voice based on the text. The processor (143) may identify a candidate projection area having the identified indicator among a plurality of candidate projection areas as a projection area.

[0232] According to one or more embodiments, the processor (143) may receive user input using gesture recognition. For example, the processor (143) may acquire an image of the user using the sensor (120), analyze the image, and obtain three-dimensional coordinate values ​​of a first point and a second point of the user's body part. In this case, the first point may be the user's elbow, and the second point may be the user's wrist.

[0233] And, the processor (143) can identify a vector corresponding to a body part of the user based on the three-dimensional coordinate values ​​of the first point and the second point. For example, the processor (143) can obtain a vector based on the three-dimensional coordinate values ​​of the first point and the second point. At this time, the starting point of the vector may be the three-dimensional coordinate value of the first point, and the ending point of the vector may be the three-dimensional coordinate value of the second point. And, the processor (143) can identify a candidate projection area in the direction toward which the vector is directed among the plurality of candidate projection areas based on the three-dimensional coordinate values ​​of the vector and the plurality of candidate projection areas as the projection area. For example, the processor (143) can identify a point where the vector intersects a plane around the user, and identify a distance between the identified point and the center point of each of the plurality of candidate projection areas. Based on the identified distances, the processor (143) can identify a candidate projection area having the shortest distance among the plurality of candidate projection areas as the candidate projection area in the direction toward which the vector is directed.

[0234] The processor (143) can project image content onto a projection area using a projector (110) (S290).

[0235] A projection area may include an area on which image content provided to a user is projected.

[0236] For example, the processor (143) can control the projector (110) to project image content corresponding to user input onto the projection area.

[0237] According to one or more embodiments, when a voice signal for a user's voice is received, the processor (143) may convert the voice signal into text and analyze the meaning of the text to obtain information about the user's intent. The processor (143) may obtain image content corresponding to the user's intent and control the projector (110) to project the image content onto a projection area.

[0238] For example, if the user's intention is related to searching (or requesting) image content, the processor (143) may analyze text to obtain keywords and search for image content related to the keywords. At this time, the processor (143) may search for image content related to the keywords among a plurality of image contents stored in the memory (142). In addition, the processor (143) may obtain image content related to the keywords from a server providing a streaming service. In addition, the processor (143) may request a search for the keywords from websites, search engines, etc. to obtain image content related to the keywords.

[0239] For example, image content may include various image content such as television programs, movies, or dramas. In addition, image content may include various information such as weather, time, etc., or various information and advertising videos related to services provided within the space where the robot (100) is located.

[0240] In this way, the robot (100) can receive user input through interaction with the user and project image content corresponding to the user input onto the user's surroundings. In the present disclosure, the user may also be represented as an interaction target.

[0241] For example, the processor (143) can use a transformation matrix to align image content to an area corresponding to a projection area in an image to be projected by the projector (110) and control the projector (110) to project an image including the image content. At this time, when the image is projected, the image content included in the image can be projected onto the projection area.

[0242] For example, it is assumed that a first candidate projection area (e.g., 512 in FIG. 5B) among a plurality of candidate projection areas is determined to be a projection area. The processor (143) can use the transformation matrix (H1) to align an area corresponding to the first candidate projection area in an image to be projected by the projector (110). In addition, referring to FIG. 11, the processor (143) can control the projector (110) to project an image (1110) including image content (1111). At this time, when the image (1110) is projected, the image content (1111) can be located in the first candidate projection area (1120).

[0243] Meanwhile, in the above-described example, when sequentially displaying information about the priority of multiple candidate projection areas in multiple candidate projection areas, an image including an indicator such as a number or letter was projected, but the present invention is not limited thereto.

[0244] The processor (143) can control the projector (110) to sequentially project images according to user input.

[0245] At this time, the image may include a sub-image projected onto the candidate projection area. The sub-image may be monochrome and may not include an indicator corresponding to the priority.

[0246] For example, the processor (143) may identify a candidate projection area with the highest priority based on the priorities of multiple candidate projection areas, and control the projector (110) to project an image corresponding to the identified candidate projection area. For example, referring to FIG. 12A, the processor (143) may control the projector (110) to project an image (1210) including a sub-image (1211). At this time, when the image (1210) is projected, the sub-image (1211) may be located in the first candidate projection area (1220).

[0247] The processor (143) receives user input while an image is being projected, and controls the projector (110) to project an image corresponding to the candidate projection area with the next highest priority based on the user input, or identifies the candidate projection area in which the current sub-image is projected as a projection area.

[0248] For example, when a voice signal for a user's voice is received, the processor (143) may convert the voice signal into text and analyze the meaning of the text to obtain information about the user's intent. Based on the user's intent, the processor (143) may control the projector (110) to project an image corresponding to the candidate projection area with the next highest priority, or identify the candidate projection area where the current sub-image is projected as a projection area.

[0249] For example, referring to FIG. 12B, while an image (1210) is being projected, a user may utter a voice signal such as “next” (1230). The processor (143) may receive the voice signal and convert the voice signal into text to determine the intent of the user’s voice. The processor (143) may identify that the user’s voice signal such as “next” (1230) includes an intent to request that an image corresponding to a candidate projection area with the next highest priority be projected. In this case, the processor (143) may identify a candidate projection area with a second highest priority after the candidate projection area on which the current sub-image is being projected, based on the priorities of the plurality of candidate projection areas, and control the projector (110) to project an image corresponding to the identified candidate projection area. For example, referring to FIG. 12C, the processor (143) may control the projector (110) to project an image (1240) including a sub-image (1241). At this time, when the image (1240) is projected, the sub-image (1241) can be located in the second candidate projection area (1250).

[0250] For example, as shown in FIG. 12D, while an image (1210) is being projected, a user may utter a voice signal such as “here” (1260). The processor (143) may receive the voice signal and convert the voice signal into text to determine the intent of the user’s voice. The processor (143) may identify that the user’s voice signal such as “here” (1260) includes an intent to select a candidate projection area where the current sub-image is projected as the projection area. In this case, the processor (143) may identify the first candidate projection area (1220) where the sub-image (1211) is projected as the projection area.

[0251] In this way, the processor (143) can project images corresponding to candidate projection areas one by one and determine a projection area from among a plurality of candidate projection areas based on user input.

[0252] As described above, the robot (100) can identify a projection area around the user.

[0253] For example, the robot (100) can acquire information using a sensor (120) (e.g., a lidar sensor, a camera, etc.) and recognize the environment around the robot (100) by exploring the surroundings of the robot (100) using the acquired information. The robot (100) recognizing the surroundings may include the robot (100) acquiring information about the direction in which an object is located around the robot (100), the distance between the robot (100) and the object, etc. In addition, the robot (100) can detect a user from an image acquired using the sensor (120) and track the detected user. For example, the robot (100) can track the user by comparing characteristic information of the user (e.g., the user's size, color, shape, outline, etc.) between image frames, and can add an ID to each user to distinguish the user.

[0254] At this time, when a voice signal including a trigger word is received, the processor (143) can identify the user who uttered the voice signal and identify a projection area around the identified user.

[0255] The trigger word may include a word for calling the robot (100). The trigger word may be a preset word or a word selected by user input from among a plurality of preset words.

[0256] For example, when a voice signal for a user's voice is received, the processor (143) can convert the voice signal into text and analyze the text to identify whether the user's voice includes a trigger word.

[0257] And, when the processor (143) identifies that the user's voice includes a trigger word, it can identify the user who uttered the voice including the trigger word and identify a projection area around the identified user.

[0258] According to one or more embodiments, the robot (100) may include a plurality of microphones. When a voice signal is received through the plurality of microphones, the processor (143) may identify the direction in which the voice signal was received to identify the user who uttered the voice. For example, the processor (143) may identify the direction of the sound source using sound localization and identify the user located in the identified direction as the user who uttered the trigger word. In addition, the processor (143) may identify a projection area around the user using an image including the user who uttered the trigger word. In this case, the processor (143) may control the driving unit (130) to rotate the robot (100) so that the field of view of the robot (100) (e.g., the field of view of the sensor (120)) is directed toward the user who uttered the trigger word, and may also acquire an image including the user who uttered the trigger word using the sensor (120).

[0259] According to one or more embodiments, the processor (143) may identify a user who uttered the trigger word based on an image acquired using the sensor (120).

[0260] For example, the processor (143) can detect the outline of an object in an image and calculate a probability value indicating the degree to which the shape of a pre-stored object (e.g., a mouth) matches the detected outline. Then, the processor (143) can identify the object having the highest probability value among the probability values ​​calculated for the outline of the object as the user's mouth. In this case, when the processor (143) identifies that a voice signal including a trigger word has been received, the processor (143) can identify a user whose mouth shape changes in the image frames acquired at the time when the voice signal was received (or a time interval from a time point before a preset time point from the time when the voice signal was received to the time when the voice signal was received) and identify the identified user as the user who uttered the trigger word. Then, the processor (143) can identify a projection area around the user by using an image including the user who uttered the trigger word.

[0261] FIG. 13 is a block diagram illustrating a configuration of a robot according to one or more embodiments.

[0262] Referring to FIG. 13, the robot (100) may include a projector (110), a sensor (120), a driving unit (130), a main module (140), an input interface (150), and an output interface (160). However, this configuration is exemplary, and new configurations may be added. Meanwhile, since a detailed description of the configuration may overlap with the configuration illustrated in FIG. 1A, additional implementation details of one or more embodiments illustrated in FIG. 13 may refer to FIG. 1A and its description.

[0263] The sensor (120) can detect spatial structures or objects. Objects may include walls and obstacles within the space. Obstacles may include various objects present in the space, such as furniture, home appliances, remote controls, keys, people, pets, etc. Additionally, information acquired by the sensor (120) can be used to create a map of the space.

[0264] The sensor (120) may include a lidar sensor, an obstacle detection sensor, and a driving detection sensor. The lidar sensor outputs a laser in a 360-degree direction, and when a laser reflected from an object is received, the lidar sensor analyzes the time difference taken for the laser to reflect from the object and return, the signal intensity of the received laser, etc., to obtain geometry information about the space. The geometry information may include the position, distance, direction, etc. of the object. The lidar sensor may provide the obtained geometry information to the processor (143).

[0265] The obstacle detection sensor can detect obstacles around the robot. For example, the obstacle detection sensor may include at least one of an ultrasonic sensor, an infrared sensor, an RF (radio frequency) sensor, a geomagnetic sensor, and a PSD (Position Sensitive Device) sensor. The obstacle detection sensor can detect obstacles present in front, behind, to the side, or along the robot's movement path. The obstacle detection sensor can provide information on the detected obstacles to the processor (143).

[0266] The driving detection sensor can detect the driving of the robot (100). For example, the driving detection sensor can include at least one of a gyro sensor, a wheel encoder, and an acceleration sensor. The gyro sensor can detect the rotation direction and rotation angle of the robot (100). The wheel encoder can detect the number of rotations of the wheels of the robot (100). The acceleration sensor can detect changes in the speed of the robot (100). The driving detection sensor can provide detected driving information to the processor (143).

[0267] For example, the processor (143) may generate a map of a space using information acquired through the sensor (120). The map may be generated during an initial exploration of the space. For example, the processor (143) may explore the space using a lidar sensor, acquire topographic information about the space, and generate a map of the space using the topographic information. In this case, the map may include a grid map.

[0268] Additionally, the processor (143) can identify the location of the robot (100) on the map using SLAM (Simultaneous Localization and Mapping).

[0269] For example, the processor (143) may acquire spatial topographic information using a lidar sensor, compare the acquired topographic information with pre-stored topographic information, or identify the location of the robot (100) on the map by comparing the acquired topographic information. However, the present disclosure is not limited to the above-described example, and for example, the processor (143) may also identify the location of the robot (100) on the map through SLAM using a camera.

[0270] Additionally, the processor (143) can control the movement of the robot (100) using information acquired through the sensor (120).

[0271] For example, the processor (143) can control the driving unit (130) to allow the robot (100) to drive in a space using a map stored in the memory (142). In addition, the processor (143) can obtain information using the sensor (120) while the robot (100) drives in a space, and can detect obstacles around the robot (100) using the obtained information. When an obstacle is detected, the processor (143) can determine a driving pattern of the robot (100), such as straight forward or turning, and control the driving unit (130) to allow the robot (100) to drive while avoiding the obstacle according to the determined driving pattern. In addition, the processor (143) can identify driving information, such as the moving speed of the robot (100) and the distance traveled by the robot cleaner (100), using the information obtained by the sensor (120), and can update the location of the robot (100) on the map based on the driving information.

[0272] The input interface (150) includes circuitry. The input interface (150) can receive user input and transmit the user input to the processor (143). For example, the input interface (160) can receive various user inputs for setting or selecting various functions supported by the robot (100).

[0273] The input interface (150) may include various types of input devices.

[0274] According to one or more embodiments, the input interface (160) may include a physical button. The physical button may include a function key or a dial button. The physical button may also be implemented as one or more keys.

[0275] According to one or more embodiments, the input interface (150) may receive user input using a touch method. For example, the input interface (150) may be implemented as a touch screen capable of performing the function of a display (161).

[0276] According to one or more embodiments, the input interface (150) may receive a voice signal corresponding to a user's voice using a microphone. In this case, the input interface (150) may include one or more microphones. The processor (143) may perform a function corresponding to the user's voice using the voice signal.

[0277] The output interface (160) may include a display (161) and a speaker (162).

[0278] The display (161) can display various screens. The processor (143) can display various notifications, messages, information, etc. related to the operation of the robot (100) on the display (161).

[0279] The display (161) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, the display (161) may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes) display, a micro LED display, a Mini LED display, a QLED (Quantum dot light-emitting diodes) display, etc.

[0280] The speaker (162) can output audio signals. The processor (143) can output warning sounds, notification messages, response messages corresponding to user input, etc. related to the operation of the robot (100) through the speaker (162).

[0281] FIG. 14 is a flowchart illustrating a method of projecting an image of a robot including a projector according to one or more embodiments.

[0282] The robot senses the user's surroundings and identifies multiple candidate projection areas based on the acquired information (S1410).

[0283] The robot identifies priorities of multiple candidate projection areas (S1420).

[0284] The robot controls the projector to project an image onto an area including a plurality of candidate projection areas based on the positions of the plurality of candidate projection areas and the priorities of the plurality of candidate projection areas, thereby simultaneously displaying information on the priorities of the plurality of candidate projection areas onto the plurality of candidate projection areas or sequentially displaying information on the priorities of the plurality of candidate projection areas onto the plurality of candidate projection areas (S1430).

[0285] The robot identifies a candidate projection area selected from among multiple candidate projection areas based on user input as a projection area (S1440).

[0286] The robot projects image content onto the projection area using a projector (S1450).

[0287] In operation S1430, the robot can identify a plurality of regions corresponding to the plurality of candidate projection regions from images to be projected on the regions based on the positions of the plurality of candidate projection regions, and control the projector to project images included in the plurality of regions as a plurality of sub-images. Each of the plurality of sub-images can include an indicator corresponding to the priority of each of the plurality of candidate projection regions. The indicator can include a number indicating the priority of the candidate projection region.

[0288] In operation S1430, the robot can identify an area corresponding to a candidate projection area from an image to be projected on an area based on the location of the candidate projection area for each candidate projection area, and control the projector so that each sub-image sequentially projects a plurality of images included in each area. Each sub-image can include an indicator corresponding to the priority of each of the plurality of candidate projection areas. The indicator can include a number indicating the priority of the candidate projection area.

[0289] In operation S1420, the robot can identify priorities of the plurality of candidate projection areas based on the size of each of the plurality of candidate projection areas and the distance between the user and each of the plurality of candidate projection areas.

[0290] In operation S1410, the robot senses the user's surroundings using a sensor, generates a three-dimensional map of the user's surroundings based on the acquired information, identifies a plane in the user's surroundings based on the three-dimensional map, identifies a plurality of areas on the identified planes whose aspect ratios are equal to the aspect ratio of the projection image, and identifies a plurality of candidate projection areas among the plurality of areas based on characteristics of the plurality of areas.

[0291] In operation S1410, if a region among the plurality of regions has a saturation higher than a threshold value based on RGB values ​​of points included in the plurality of regions, the robot can identify the remaining regions, excluding the identified region among the plurality of regions, as multiple candidate projection regions.

[0292] In operation S1410, the robot can acquire an image using a sensor, identify a rotation angle range of the sensor based on a location of a user identified in the image and a field of view of the sensor, and acquire information using the sensor while the sensor rotates within the rotation angle range.

[0293] In this case, the robot can obtain a bounding box for the user from the acquired image, identify the pixel distance between the center pixel of the acquired image and the pixel of the bounding box, and identify the rotational angle range of the sensor based on the focal length and pixel distance of the sensor.

[0294] Meanwhile, according to one or more embodiments, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When an instruction is executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instruction using other components. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0295] Additionally, according to one or more embodiments, the methods according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0296] Additionally, according to one or more embodiments, the various embodiments described above may be implemented in a computer-readable recording medium using software, hardware, or a combination thereof, or a computer or similar device. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments, such as the procedures and functions described herein, may be implemented as separate software. Each software may perform one or more functions and operations described herein.

[0297] Meanwhile, computer instructions for performing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in such a non-transitory computer-readable medium are executed by the processor of the device, the device performs processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by the device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0298] Additionally, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and other sub-components may be further included in the various embodiments. Some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions performed by each respective component prior to integration. Operations performed by modules, programs, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, or other operations may be added.

[0299] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In robots, projector; sensor; Memory for storing instructions; and comprising one or more processors; The above instructions, when executed by the at least one processor, cause the robot to: Identifying multiple candidate projection areas based on first information acquired by sensing the user's surroundings through the above sensor, Identify the priorities of the above multiple candidate projection areas, Controlling the projector to project a first image on an area including the plurality of candidate projection areas based on the plurality of positions of the plurality of candidate projection areas and the priorities, and display second information about the priorities on the plurality of candidate projection areas; Identifying a candidate projection area selected from among the above multiple candidate projection areas based on user input as a projection area, A robot that projects image content onto the projection area via the above projector.

2. In paragraph 1, The one or more processors execute the instructions so that the robot, Identifying a plurality of regions corresponding to the plurality of candidate projection regions in the second image to be projected onto the region based on the plurality of positions, Controlling the above projector to project the second image, A robot wherein the second image includes a plurality of sub-images corresponding to the plurality of areas, and the plurality of sub-images include a plurality of indicators corresponding to the priorities.

3. In paragraph 2, A robot wherein the above plurality of indicators include a plurality of numbers representing the priorities.

4. In paragraph 1, The one or more processors execute the instructions so that the robot, Identifying a plurality of regions corresponding to the plurality of candidate projection regions based on the plurality of locations, Controlling the above projector to sequentially project a plurality of sub-images corresponding to the plurality of areas, A robot wherein the plurality of sub-images include a plurality of indicators corresponding to the priorities.

5. In paragraph 4, A robot wherein the above plurality of indicators include a plurality of numbers representing the priorities.

6. In paragraph 1, The one or more processors execute the instructions so that the robot, A robot that identifies the priority based on a plurality of sizes of the plurality of candidate projection areas and a plurality of distances between the user and the plurality of candidate projection areas.

7. In paragraph 1, The one or more processors execute the instructions so that the robot, Generate a three-dimensional map of the surroundings based on the first information, Identifying a plane in the surroundings based on the above three-dimensional map, Identifying a plurality of regions having a first aspect ratio that matches a second aspect ratio of the projection image on the above plane, A robot that identifies a plurality of candidate projection areas among the plurality of areas based on characteristics of the plurality of areas.

8. In paragraph 7, The one or more processors execute the instructions so that the robot, Identify the remaining areas, excluding the identified areas among the above multiple areas, as the multiple candidate projection areas, The saturation of the above identified area is greater than a preset threshold value, A robot wherein the identified area is determined based on RGB values ​​of a plurality of points in the plurality of areas.

9. In paragraph 1, The one or more processors execute the instructions so that the robot, Obtain a third image through the above sensor, Identify the location of the user in the third image above, Identify the rotational angle range of the sensor based on the user's location and the field of view of the sensor, A robot that acquires the first information through the sensor while the sensor rotates within the rotation angle range.

10. In paragraph 9, The one or more processors execute the instructions so that the robot, Obtain a bounding box for the user based on the third image, Identify the pixel distance between the center pixel of the third image and the pixel of the bounding box, A robot that identifies the rotation angle range based on the focal length of the sensor and the pixel distance.

11. A method for projecting an image of a robot including a projector, A step of identifying a plurality of candidate projection areas based on first information acquired by sensing the user's surroundings through a sensor; A step of identifying priorities of the plurality of candidate projection areas; A step of projecting a first image through the projector onto an area including the plurality of candidate projection areas based on the plurality of locations of the plurality of candidate projection areas and the priorities; A step of displaying second information about the above priorities in the plurality of candidate projection areas; A step of identifying a candidate projection area selected from among the plurality of candidate projection areas based on user input as a projection area; and An image projection method comprising the step of projecting image content onto the projection area through the projector.

12. In paragraph 11, The steps indicated above are: A step of identifying a plurality of regions corresponding to the plurality of candidate projection regions in a second image to be projected onto the region based on the plurality of positions; and a step of controlling the projector to project the second image; An image projection method, wherein the second image includes a plurality of sub-images corresponding to the plurality of areas, and the plurality of sub-images include a plurality of indicators corresponding to the priorities.

13. In paragraph 12, An image projection method wherein the plurality of indicators include a plurality of numbers representing the priorities.

14. In paragraph 11, The steps indicated above are: A step of identifying a plurality of regions corresponding to the plurality of candidate projection regions based on the plurality of locations; and A step of controlling the projector to sequentially project a plurality of sub-images corresponding to the plurality of areas; An image projection method, wherein the plurality of sub-images include a plurality of indicators corresponding to the priorities.

15. In paragraph 14, An image projection method wherein the plurality of indicators include a plurality of numbers representing the priorities.

Images