Robot for projecting image and method for projecting image thereof

Abstract

A robot includes: a projector; a sensor; a memory storing instructions; and at least one processor. The instructions, when executed by the at least one processor, cause the robot to: identify a plurality of candidate projection areas based on first information obtained by sensing a surrounding environment of a user via the sensor; identifying priorities of the plurality of candidate projection areas; projecting a first image at a region including the plurality of candidate projection regions based on a plurality of positions of the plurality of candidate projection regions and the priority; controlling the projector to display second information about the priority at the plurality of candidate projection areas; identifying, as a projection area, a candidate projection area selected from among the plurality of candidate projection areas based on a user input; and projecting image content at the projection area via the projector.

Description

Technical Field

[0001] This disclosure relates to a robot that projects images and a method for projecting images. Background Technology

[0002] In addition to repetitive functions, robots can autonomously move by detecting their surroundings and collecting information in real time using sensors, cameras, and other means.

[0003] The robots described above are currently used in many fields and provide a variety of services through interaction with users. Summary of the Invention

[0004] According to one aspect of this disclosure, a robot includes: a projector; a sensor; a memory storing instructions; and one or more processors. The instructions, when executed by the one or more processors, cause the robot to: identify a plurality of candidate projection regions based on first information obtained by sensing the user's surrounding environment via the sensor; identify a priority order of the plurality of candidate projection regions; control the projector to: project a first image onto a region including the plurality of candidate projection regions based on a plurality of locations of the plurality of candidate projection regions and the priority order, and display second information regarding the priority order at the plurality of candidate projection regions; identify a candidate projection region selected from the plurality of candidate projection regions based on user input as a projection region; and project image content onto the projection region via the projector.

[0005] The one or more processors may be configured to execute the instructions to cause the robot to: identify, based on the plurality of positions, a plurality of regions of a second image to be projected into the region, the plurality of regions corresponding to the plurality of candidate projection regions; and control the projector to project the second image. The second image may include a plurality of sub-images corresponding to the plurality of regions, and the plurality of sub-images may include a plurality of indicators corresponding to the priority order.

[0006] The plurality of indicators may include multiple numbers indicating priority order.

[0007] The one or more processors may be configured to execute the instructions to cause the robot to: identify multiple regions corresponding to the multiple candidate projection regions based on the multiple locations; and control the projector to sequentially project multiple sub-images corresponding to the multiple regions. The multiple sub-images may include multiple indicators corresponding to the priority order.

[0008] The plurality of indicators may include a plurality of numbers indicating the priority order.

[0009] One or more processors may be configured to execute the instructions to enable the robot to identify the priority order based on: multiple sizes of the multiple candidate projection areas, and multiple distances between the user and the multiple candidate projection areas.

[0010] One or more processors may be configured to execute the instructions to cause the robot to: generate a three-dimensional map of the surrounding environment based on first information; identify a plane from the surrounding environment based on the three-dimensional map; identify a plurality of regions on the plane having a first aspect ratio, the first aspect ratio matching a second aspect ratio of a projected image; and identify a plurality of candidate projection regions from the plurality of regions based on features of the plurality of regions.

[0011] One or more processors may be configured to execute the instructions to cause the robot to identify the remaining regions in the plurality of regions, excluding the identified regions, as the plurality of candidate projection regions. The saturation of the identified regions may be greater than or equal to a threshold, and the identified regions may be determined based on the RGB values ​​of multiple points in the plurality of regions.

[0012] One or more processors may be configured to execute the instructions to cause the robot to: acquire a third image via the sensor; identify the position of the user in the third image; identify the rotation angle range of the sensor based on the user's position and the sensor's field of view; and acquire first information via the sensor while the sensor rotates within the rotation angle range.

[0013] One or more processors may be configured to execute the instructions to cause the robot to: obtain a bounding box for the user based on a third image; identify the pixel distance between the center pixel of the third image and the pixel of the bounding box; and identify the rotation angle range based on the focal length of the sensor and the pixel distance.

[0014] According to one aspect of this disclosure, a method for projecting an image via a robot including a projector includes: identifying a plurality of candidate projection regions based on first information obtained by sensing the user's surrounding environment via sensors; identifying a priority order of the plurality of candidate projection regions; projecting a first image via the projector onto a region including the plurality of candidate projection regions based on a plurality of locations of the plurality of candidate projection regions and the priority order; displaying second information regarding the priority order at the plurality of candidate projection regions; identifying a candidate projection region selected from the plurality of candidate projection regions based on user input as a projection region; and projecting image content via the projector onto the projection region.

[0015] The display step may include: identifying multiple regions of a second image to be projected into the region based on the multiple locations, the multiple regions corresponding to the multiple candidate projection regions; and controlling the projector to project the second image. The second image may include multiple sub-images corresponding to the multiple regions, and the multiple sub-images may include multiple indicators corresponding to the priority order.

[0016] The plurality of indicators may include a plurality of numbers indicating the priority order.

[0017] The display steps may include: identifying multiple regions corresponding to the multiple candidate projection regions based on the multiple locations; and controlling the projector to sequentially project multiple sub-images corresponding to the multiple regions. The multiple sub-images may include multiple indicators corresponding to the priority order.

[0018] The plurality of indicators may include a plurality of numbers indicating the priority order.

[0019] The step of identifying the priority order may include identifying the priority order based on the following: multiple sizes of the plurality of candidate projection regions, and multiple distances between the user and the plurality of candidate projection regions.

[0020] The step of identifying the plurality of candidate projection regions may include: generating a three-dimensional map of the surrounding environment based on first information; identifying a plane from the surrounding environment based on the three-dimensional map; identifying a plurality of regions on the plane having a first aspect ratio, the first aspect ratio matching a second aspect ratio of the projected image; and identifying the plurality of candidate projection regions from the plurality of regions based on the features of the plurality of regions.

[0021] The step of identifying the plurality of candidate projection regions may include: identifying the remaining regions in the plurality of regions other than the identified regions as the plurality of candidate projection regions. The saturation of the identified regions may be greater than or equal to a threshold, and the identified regions may be determined based on the RGB values ​​of multiple points in the plurality of regions.

[0022] The step of obtaining the first information may include: obtaining a third image via the sensor; identifying the position of the user in the third image; identifying the rotation angle range of the sensor based on the user's position and the field of view of the sensor; and obtaining the first information via the sensor while the sensor rotates within the rotation angle range.

[0023] According to one aspect of this disclosure, a non-transitory computer-readable recording medium having instructions recorded thereon, which, when executed by one or more processors of a robot including a projector, cause the robot to: identify a plurality of candidate projection regions based on first information obtained by sensing the user's surrounding environment via sensors; identify a priority order of the plurality of candidate projection regions; control the projector to: project a first image onto a region including the plurality of candidate projection regions based on a plurality of locations of the plurality of candidate projection regions and the priority order, and display second information regarding the priority order at the plurality of candidate projection regions; identify a candidate projection region selected from the plurality of candidate projection regions based on user input as a projection region; and project image content onto the projection region via the projector. Attached Figure Description

[0024] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which: Figure 1a and Figure 1b This is a block diagram illustrating the configuration of a robot according to one or more embodiments; Figure 2 This is a flowchart illustrating a method for projecting images through a robot according to one or more embodiments; Figure 3 This is an illustration showing examples of user-oriented bounding boxes detected from an image according to one or more embodiments; Figure 4 This is a diagram illustrating an example of a method for a robot, according to one or more embodiments, to sense the area surrounding a user; Figure 5a and Figure 5b This is a diagram illustrating an example of an operation in which a robot, according to one or more embodiments, is configured to identify multiple candidate projection regions; Figure 6a , Figure 6b , Figure 6c and Figure 6d This is an illustration of an example of a method for locating a candidate projection region within a pixel range of an image, according to one or more embodiments; Figure 7 This is an illustration of an example of a method for a robot, according to one or more embodiments, to generate an image that will be projected using a projector; Figure 8 This is a diagram illustrating an example of a robot configured to project images according to one or more embodiments; Figure 9This is an illustration of an example of a method for a robot, according to one or more embodiments, to generate multiple images that will be projected using a projector; Figure 10a , Figure 10b , Figure 10c and Figure 10d This is a diagram illustrating an example of a robot configured to project multiple images according to one or more embodiments; Figure 11 This is a diagram illustrating an example of the operation of a robot configured to project image content at a projection area according to one or more embodiments; Figure 12a , Figure 12b , Figure 12c and Figure 12d This is a diagram illustrating an example of the operation of a robot configured to project multiple images according to one or more embodiments; Figure 13 This is a block diagram illustrating the configuration of a robot according to one or more embodiments; and Figure 14 This is a flowchart illustrating a method for projecting images through a robot according to one or more embodiments. Detailed Implementation

[0025] The embodiments described in this disclosure and the configurations shown in the accompanying drawings are examples of embodiments, and various modifications may be made without departing from the scope and spirit of this disclosure.

[0026] The terminology used in this disclosure will be briefly described, and the disclosure will be described in detail. In this disclosure, the expression "at least one of a, b, or c" may refer to "a", "b", "c", "a and b", "a and c", "b and c", "all of a, b, and c", or variations thereof.

[0027] The terms used in this disclosure are those currently widely used and chosen in consideration of their function herein. However, terms may change depending on the intent of a person skilled in the art, legal or technical interpretation, the emergence of new technologies, etc. Furthermore, in some cases, arbitrarily chosen terms may exist, and in such cases, the meaning of the term will be described in more detail in the corresponding description. Therefore, the terms used herein should not be understood as their names, but rather based on their meaning and the overall context of this disclosure.

[0028] Unless otherwise stated, singular expressions include plural expressions. The terms used herein (including technical or scientific terms) may have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Furthermore, terms including ordinal numbers such as “first” or “second” as used herein may be used to describe various elements, but these elements are not limited by these terms. These terms may be used only for the purpose of distinguishing one element from another.

[0029] Throughout this disclosure, when a part is described as "including" another part, unless otherwise indicated, the foregoing may mean that another element may be further included rather than excluded. Furthermore, terms such as "component" or "module" described in this disclosure may refer to a unit that performs at least one function or operation, and that unit may be implemented in hardware or software, or a combination of hardware and software.

[0030] The term “and / or” may include a combination of the multiple related elements described or any one of the multiple related elements described.

[0031] Furthermore, various elements and areas have been schematically shown in the accompanying drawings. Therefore, the technical spirit of this disclosure is not limited to the relative dimensions and distances shown in the drawings.

[0032] In this disclosure, text can be obtained based on speech signals corresponding to a user's speech, and the process of obtaining information about the user's intent based on the text can be performed by an artificial intelligence model. The artificial intelligence model can be implemented as a built-in device included in a robot. However, this is not the only possibility, and the artificial intelligence model can be stored in a server connected to the robot. If the artificial intelligence model is stored in the server, the robot can send speech signals corresponding to the user's speech to the server and receive information about the user's intent or control commands based on the user's intent from the server.

[0033] This disclosure relates to a robot configured to determine a projection area from the user's surrounding environment by considering features of the user's surrounding environment, project an image onto the projection area, and provide the user with various image content, as well as a method for projecting images through the robot.

[0034] The embodiments will now be described in detail with reference to the accompanying drawings to assist those skilled in the art in understanding the subject matter. However, this disclosure can be implemented in various different forms, and it should be noted that this disclosure is not limited to the embodiments described herein. Furthermore, in the drawings, the same reference numerals may be used to indicate the same elements.

[0035] This disclosure will now be described with reference to the accompanying drawings.

[0036] Figure 1a and Figure 1b This is a block diagram illustrating the configuration of a robot according to one or more embodiments.

[0037] Reference Figure 1a The robot 100 may include a projector 110, a sensor 120, a drive unit 130, and a main module 140. Furthermore, Figure 1a The configuration of the robot 100 shown is exemplary, and some configurations can be added according to embodiments.

[0038] Robot 100 according to one or more embodiments may be a mobile robot (e.g., a mobile robot). Robot 100 may also be referred to as an autonomous mobility device, a mobile device, etc., but will be described as robot 100 in this disclosure. Movement of robot 100 may include detecting the position of the robot and obstacles by exploring the surrounding environment and using the detected information to move autonomously within a space. The space in which robot 100 moves may include various indoor spaces in which robot 100 can move, such as, but not limited to, homes, offices, hotels, factories, shops, supermarkets, restaurants, etc.

[0039] The robot 100, according to one or more embodiments, can be implemented as various types of robots. For example, the robot 100 can be implemented as a robotic cleaner that performs cleaning while moving within a space, a guide robot that guides users within a space or provides various information related to the services provided within the space, or a transport robot or service robot that transports loaded products to a location within the space, or a mobile projection device that can project images while moving between locations, etc.

[0040] Projector 110 can project images. Images can include still images and moving images (e.g., video). Moving images can include various visual information indicating the movement of an object using multiple consecutive still images. In this case, each of the multiple still images included in the video can represent a frame (or image frame).

[0041] Projector 110 can project an image onto a projection surface using light emitted from a light source. For example, projector 110 can project images using a cathode ray tube (CRT) method, a liquid crystal display (LCD) method, a digital light processing (DLP) method, or a liquid crystal on silicon (LCoS) method. The projection surface can be a separately provided screen, but is not limited to this, and can be various walls, a surface of an object, etc., within the space in which robot 100 moves.

[0042] Sensor 120 can be configured to sense various types of information. One or more processors 143 can acquire various types of information based on the sensed values ​​from sensor 120. For example, the information acquired by sensor 120 may include image and depth information. The image may include the RGB values ​​of each of a plurality of pixels included in the image. The depth information may include a depth map, wherein the depth map includes the depth value of each of the plurality of pixels.

[0043] According to one or more embodiments, sensor 120 may include a stereo camera. Sensor 120 may include an RGB-D camera. Sensor 120 may include an RGB camera and a light detection and ranging (LiDAR) sensor. However, embodiments are not limited thereto, and sensor 120 may include various sensors that can be used to obtain image and depth information.

[0044] The drive unit 130 can control the movement of the robot 100. For example, the drive unit 130 can move the robot 100, or stop the robot 100 in motion, and control the movement speed and / or direction of the robot 100.

[0045] For example, the movement type of robot 100 can be either wheeled or walking.

[0046] The wheel type can indicate the method by which robot 100 moves by rotating wheels. If robot 100 is a wheeled robot, then robot 100 may include one or more wheels. Drive unit 130 may include means for generating power to rotate the wheels. For example, depending on the fuel (or energy) used, drive unit 130 may be implemented as a gasoline engine, diesel engine, liquefied petroleum gas (LPG) engine, electric motor, etc.

[0047] The locomotion type can refer to the method by which robot 100 moves by the movement of its legs. If robot 100 is a locomotion type (e.g., a bipedal robot, a tripedal robot, a quadrupedal robot, etc.), then robot 100 may include two or more legs supporting robot 100. The legs may include multiple links and joints connecting the links. Drive unit 130 may include means for generating power to raise or lower the legs by rotating the links based on the joints. For example, drive unit 130 may be implemented as a motor and / or actuator.

[0048] Additionally, the drive component 130 can control the movement of a part of the robot 100. The drive component 130 can be coupled between a first part (e.g., body) and a second part (e.g., head, arm, etc.) of the robot 100. The drive component 130 can cause the second part to rotate. For example, the drive component 130 can be implemented as a motor and / or actuator.

[0049] The main module 140 can be implemented as hardware and may include a communication interface 141, a memory 142, one or more processors 143 and control units 144.

[0050] The communication interface 141 can perform data communication with electronic devices under the control of one or more processors 143. For example, the communication interface 141 may include communication circuitry that can utilize at least one of the following data communication methods to perform data communication between the robot 100 and the electronic devices: wired LAN, wireless LAN, Wi-Fi, Bluetooth, Wi-Fi Direct (WFD), infrared communication (Infrared Data Association (IrDA)), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), Global Microwave Access Interoperability (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0051] The memory 142 may store instructions, data structures, and program code that can be read by one or more processors 143. Operations performed by one or more processors 143 can be implemented by executing the instructions or code of the program stored in the memory 142.

[0052] The memory 142 may include flash memory, hard disk memory, micro multimedia card and card-type memory (e.g., SD or XD memory), non-volatile memory including at least one of read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk and optical disk, and volatile memory such as random access memory (RAM) or static random access memory (SRAM).

[0053] Memory 142 may store one or more instructions and / or programs for robot 100 to determine a projection area from the user's surrounding environment and to operate to project an image onto the projection area. For example, refer to Figure 1b The memory 142 may store instructions and / or programs for implementing the functions of the candidate projection area identification module 21, the priority order determination module 22, and the projection module 23. Figure 1b Modules 21, 22, and 23 shown can be configured to be implemented by one or more processors 143 that execute programs or instructions stored in memory 142. Therefore, the operations described below as being performed by modules 151, 152, and 153 can actually be performed by one or more processors 143.

[0054] One or more processors 143 can control the overall operation of robot 100. For example, one or more processors 143 can control the overall operation of robot 100 for determining a projection area from the user's surrounding environment and projecting an image onto the projection area by executing one or more instructions of a program stored in memory 142. For example, one or more processors 143 can perform various calculations for robot 100 to determine a projection area from the user's surrounding environment and project an image onto the projection area, and transmit signals related to the calculation results to control unit 144.

[0055] Control unit 144 can control the components of robot 100. Control unit 144 can control the components of robot 100 (e.g., projector 110, sensor 120, drive unit 130, etc.) based on signals provided from one or more processors 143. For example, control unit 144 can use signals provided from one or more processors 143 to generate control signals and provide these control signals to the components of robot 100. Therefore, the components of robot 100 can perform operations corresponding to the calculation results of one or more processors 143. Control unit 144 can be implemented as one or more ICs (e.g., controller ICs).

[0056] One or more processors 143 may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. One or more processors 143 may control one or more other elements of the robot 100 or other random combinations of elements and perform operations associated with communication or data processing. One or more processors 143 may execute one or more programs or instructions stored in memory 142. For example, one or more processors 143 may perform a method according to one or more embodiments by executing one or more instructions stored in memory 142.

[0057] When a method according to one or more embodiments includes multiple operations, the multiple operations may be executed by a single processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be executed by a first processor, or the first operation and the second operation may be executed by a first processor (e.g., a CPU), and the third operation may be executed by a second processor (e.g., a dedicated AI processor).

[0058] One or more processors 143 may be implemented as a single-core processor including one core, or as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). If one or more processors 143 are implemented as multi-core processors, each of the multiple cores included in the multi-core processor may include internal processor memory, such as cache memory and on-chip memory, and a common cache shared by the multiple cores may be included in the multi-core processor. Furthermore, each (or a portion of the multiple cores) included in the multi-core processor may independently read and execute program instructions for implementing the methods according to one or more embodiments, or all (or a portion of) the multiple cores may be interconnected to read and execute program instructions for implementing the methods according to one or more embodiments.

[0059] When a method according to one or more embodiments includes multiple operations, the multiple operations may be executed by one core of a multi-core processor, or by multiple cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, the first operation, the second operation, and the third operation may all be executed by the first core of the multi-core processor, or the first operation and the second operation may be executed by the first core of the multi-core processor, and the third operation may be executed by the second core of the multi-core processor.

[0060] In one or more embodiments, a processor may refer to a system-on-a-chip (SoC), a single-core processor, or a multi-core processor that integrates one or more processors and other electronic components, or a core included in a single-core processor or multi-core processor. The core herein may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, machine learning accelerator, etc., but one or more embodiments are not limited thereto.

[0061] For ease of description below, one or more processors 143 may be described as processor 143.

[0062] The processor 143 can identify multiple candidate projection regions from the user's surrounding environment by executing the candidate projection region recognition module 21. A candidate projection region can represent an area in the user's surrounding environment where an image will be projected. The candidate projection region recognition module 21 can identify flat surfaces with sufficient area, form, or color (e.g., saturation) as candidate projection regions from the surfaces of various objects (such as walls, household appliances, furniture, etc.) within the space where the robot 100 is located. For example, the candidate projection region recognition module 21 can identify planes in the user's surrounding environment and identify multiple candidate projection regions by considering the size and color of the areas within the plane and the aspect ratio of the projected image.

[0063] A more detailed description of the candidate projection region identification module 21 will be provided in [the following text is missing from the original] Figure 2 The descriptions of S210, S220, S230, S240 and S250 are further described in the text.

[0064] The processor 143 can identify the priority order of multiple candidate projection regions by executing the priority order determination module 22 and provide the user with information about the priority order of the multiple candidate projection regions.

[0065] Priority order can be information that specifies the order in which projection regions can be selected from multiple candidate projection regions. For example, when an image is projected onto candidate projection regions, priority order can indicate the order in which the projected image is visible to the user among the candidate projection regions.

[0066] The priority order determination module 22 can determine the priority order of multiple candidate projection regions based on their characteristics. These characteristics may include the size of the candidate projection region and the distance between the user and the candidate projection region.

[0067] In addition, the priority order determination module 22 can use the projector 110 to project an image indicating the priority order of each candidate projection area at each of the multiple candidate projection areas.

[0068] The image may include indicators corresponding to the priority order or candidate projection regions.

[0069] Indicators may include numbers or characters that indicate the priority order of candidate projection areas. Characters may include characters that represent numbers using a language or characters that have an order (e.g., the alphabet). Indicators may include various graphical elements that can visually provide the user with information about the priority order of candidate projection areas using numbers or characters, and may be replaced by representations such as graphical user interfaces (GUIs) or icons.

[0070] A more detailed description of the priority order determination module 22 will be available in [the following text is missing from the original] Figure 2 Further descriptions are provided in the descriptions of S260 and S270.

[0071] The processor 143 can project an image onto a projection area by executing the projection module 23. The projection area can represent the region of the image that is projected and provided to the user. The projection area can be determined from multiple candidate projection areas based on user input. The projection module 23 can use the projector 110 to project the image onto the projection area.

[0072] In Figure 2 The description of S290 further provides a more detailed description of the projection module 23.

[0073] Figure 2 This is a flowchart illustrating a method for projecting images through a robot according to one or more embodiments.

[0074] Reference Figure 2 The processor 143 can obtain information about the user's surrounding environment by using the sensor 120 to sense the user's surrounding environment (S210).

[0075] The processor 143 can obtain information by sensing the user’s surrounding environment using the sensor 120 while keeping the user within the field of view (FOV) of the sensor 120.

[0076] For example, the projection area can be pre-set such that it is located in the space where the robot 100's viewing direction (e.g., the sensor 120's viewing direction) lies. In this case, to make the image content projected by the robot 100 onto the projection area visible to the user, the projection area in the space can be determined based on the user's position. Therefore, the robot 100 can use the sensor 120 to sense the user's surrounding environment while keeping the user within the sensor 120's field of view, thereby obtaining spatial information based on the user's position.

[0077] The processor 143 can identify the rotation angle range of the sensor 120 used to sense the user's surroundings while keeping the user within the field of view of the sensor 120. Furthermore, the processor 143 can acquire information using the sensor 120 while the sensor 120 rotates within the rotation angle range.

[0078] For example, the processor 143 can use the sensor 120 to acquire an image and identify the rotation angle range of the sensor 120 based on the user's position identified from the image and the field of view of the sensor 120.

[0079] The user's location in the image can include the location of a bounding box for the user. The processor 143 can detect the user from the image and obtain the bounding box for the user.

[0080] A user's bounding box can be represented as a quadrilateral-shaped box that includes the user detected from the image. For example, such as... Figure 3 As shown, the processor 143 can use the sensor 120 to acquire an image 310 including the user 10. At this time, assuming the coordinates of the top left pixel of the image 310 are (0, 0), the position of the bounding box 320 can be represented by the x and y coordinates of the pixel 321 corresponding to the top left vertex of the bounding box 320 and the x and y coordinates of the pixel 322 corresponding to the bottom right vertex of the bounding box 320.

[0081] In addition, the processor 143 can identify the pixel distance between the center pixel of the image and the pixels of the bounding box.

[0082] The center pixel of an image can include the pixel located at the center of the image. For example, it can be assumed that the coordinates of the pixel at the top left of the image 310 obtained by sensor 120 are (0, 0), and the coordinates of the pixel at the bottom right of the image are (x, y). width y height x can be determined based on the resolution of the image 310 obtained by sensor 120. width y height At this point, the coordinates of the center pixel of the image can be (x... width / 2,y height / 2).

[0083] Pixel distance can include the Euclidean distance between two pixels. For example, the pixel distance between a pixel at coordinates (x1, y1) and a pixel at coordinates (x2, y2) can be expressed as... .

[0084] At this point, the processor 143 can identify the pixel corresponding to each of the four sides of the bounding box, and identify the pixel distance between the center pixel of the image and each of the identified pixels.

[0085] For example, processor 143 can identify the pixel corresponding to the left side of the bounding box. The left side of the bounding box can be a line connecting the pixel corresponding to the top-left vertex of the bounding box to the pixel corresponding to the bottom-left vertex of the bounding box.

[0086] The processor 143 can identify pixels in the image that have the same x-coordinate as the pixel corresponding to the top-left vertex of the bounding box and the same y-coordinate as the pixel at the center of the image as pixels corresponding to the left side of the bounding box. Furthermore, the processor 143 can identify the pixel distance between the pixel corresponding to the left side of the bounding box and the center coordinates of the image.

[0087] For example, if the coordinates of the pixel corresponding to the top left vertex of the bounding box are (x... upper_left y upper_left ), and the coordinates of the center pixel of the image are (x c y c Then the processor 143 can determine the coordinates of multiple pixels in the image as (x... upper_left y c The pixel of (x) is identified as the pixel corresponding to the left side of the bounding box. Furthermore, the processor 143 can identify (x) upper_left y c ) and (x c y cThe pixel distance between them.

[0088] Furthermore, the processor 143 can identify pixels corresponding to the right side of the bounding box. The right side of the bounding box can be a line connecting the pixel corresponding to the upper right vertex of the bounding box to the pixel corresponding to the lower right vertex of the bounding box.

[0089] The processor 143 can identify pixels in the image that have the same x-coordinate as the pixel corresponding to the lower right vertex of the bounding box and the same y-coordinate as the pixel at the center of the image as pixels corresponding to the right side of the bounding box. Furthermore, the processor 143 can identify the pixel distance between the pixel corresponding to the right side of the bounding box and the center coordinates of the image.

[0090] For example, if the coordinates of the pixel corresponding to the bottom right vertex of the bounding box are (x... lower_right y lower_right ), and the coordinates of the center pixel of the image are (x c y c Then the processor 143 can determine the coordinates of multiple pixels in the image as (x... lower_right y c The pixel of (x) is identified as the pixel corresponding to the right side of the bounding box. Furthermore, the processor 143 can identify (x) lower_right y c ) and (x c y c The pixel distance between them.

[0091] Furthermore, the processor 143 can identify pixels corresponding to the top edge of the bounding box. The top edge of the bounding box can be a line connecting the pixel corresponding to the top-left vertex of the bounding box to the pixel corresponding to the top-right vertex of the bounding box.

[0092] The processor 143 can identify pixels in an image that have the same x-coordinate as the image center pixel and the same y-coordinate as the pixel corresponding to the top left vertex of the bounding box as pixels corresponding to the top edge of the bounding box. Furthermore, the processor 143 can identify the pixel distance between the pixel corresponding to the right side of the bounding box and the center coordinate of the image.

[0093] For example, if the coordinates of the pixel corresponding to the top left vertex of the bounding box are (x... upper_left y upper_left ), and the coordinates of the center pixel of the image are (x c y c Then the processor 143 can determine the coordinates of multiple pixels in the image as (x... c y upper_leftThe pixel of (x) is identified as the pixel corresponding to the upper edge of the bounding box. Furthermore, the processor 143 can identify (x) c y upper_left ) and (x c y c The pixel distance between them.

[0094] Furthermore, the processor 143 can identify pixels corresponding to the bottom edge of the bounding box. The bottom edge of the bounding box can be a line connecting the pixel corresponding to the bottom left vertex of the bounding box to the pixel corresponding to the bottom right vertex of the bounding box.

[0095] The processor 143 can identify pixels in an image that have the same x-coordinate as the image center pixel and the same y-coordinate as the pixel corresponding to the lower right vertex of the bounding box as pixels corresponding to the bottom edge of the bounding box. Furthermore, the processor 143 can identify the pixel distance between the pixel corresponding to the bottom edge of the bounding box and the center coordinates of the image.

[0096] For example, if the coordinates of the pixel corresponding to the bottom right vertex of the bounding box are (x... lower_right y lower_right ), and the coordinates of the center pixel of the image are (x c y c Then the processor 143 can determine the coordinates of multiple pixels in the image as (x... c y lower_right The pixel of (x) is identified as the pixel corresponding to the bottom edge of the bounding box. Furthermore, the processor 143 can identify (x) c y lower_right ) and (x c y c The pixel distance between them.

[0097] Furthermore, the processor 143 can identify the rotation angle of the sensor 120 based on the identified pixel distance and the focal length of the sensor 120.

[0098] Focal length can include the distance from the principal point of the lens to the image sensor.

[0099] The rotation angle of sensor 120 may include a horizontal rotation angle range and a vertical rotation angle range.

[0100] For example, processor 143 can identify the range of horizontal rotation angles based on the first pixel distance, the second pixel distance, and the focal length.

[0101] At this point, the first pixel distance can be the pixel distance between the left-hand pixel of the bounding box and the center coordinates of the image, and the second pixel distance can be the pixel distance between the right-hand pixel of the bounding box and the center coordinates of the image.

[0102] Furthermore, the horizontal rotation angle range can include the angle range between the maximum rightward rotation angle of sensor 120 and the maximum leftward rotation angle of sensor 120. For example, the horizontal rotation angle range can include the angle range between the maximum rightward rotation angle of sensor 120 and the maximum leftward rotation angle of sensor 120 based on the capture angle of the image captured by sensor 120.

[0103] At this time, the maximum rightward rotation angle of sensor 120 may include the rotation angle of sensor 120 based on the left side of the user's bounding box being positioned at the left edge of the acquired image by rotating sensor 120 to the right. Furthermore, the maximum leftward rotation angle of sensor 120 may be the rotation angle of sensor 120 based on the right side of the user's bounding box being positioned at the right edge of the acquired image by rotating sensor 120 to the left.

[0104] For example, processor 143 can identify the maximum rightward rotation angle of sensor 120 as (sensor's horizontal FOV) / 2 + tan θ, based on the fact that the x-coordinate value of the pixel corresponding to the left of the bounding box is greater than the x-coordinate value of the center pixel of the image. -1 (First pixel distance / focal length). Furthermore, the processor 143 can identify the maximum rightward rotation angle of the sensor 120 as (sensor's horizontal FOV) / 2 - tan(x-coordinate) based on the x-coordinate value of the pixel corresponding to the left of the bounding box being less than the x-coordinate value of the center pixel of the image. -1 (First pixel distance / focal length). In addition, the processor 143 can identify the maximum rightward rotation angle of the sensor 120 as (sensor's horizontal FOV) / 2, based on the fact that the x-coordinate value of the pixel corresponding to the left of the bounding box is the same as the x-coordinate value of the pixel at the center of the image.

[0105] Furthermore, the processor 143 can identify the maximum leftward rotation angle of the sensor 120 as (sensor's horizontal FOV) / 2 - tan θ, based on the fact that the x-coordinate value of the pixel corresponding to the right side of the bounding box is greater than the x-coordinate value of the center pixel of the image. -1 (Second pixel distance / focal length). Furthermore, the processor 143 can identify the maximum leftward rotation angle of the sensor 120 as (sensor's horizontal FOV) / 2 + tan θ, based on the x-coordinate value of the pixel corresponding to the right side of the bounding box being less than the x-coordinate value of the center pixel of the image. -1 (Second pixel distance / focal length). In addition, the processor 143 can identify the maximum leftward rotation angle of the sensor 120 as (sensor's horizontal FOV) / 2, based on the fact that the x-coordinate value of the pixel corresponding to the right side of the bounding box is the same as the x-coordinate value of the pixel at the center of the image.

[0106] For example, processor 143 can identify the range of vertical rotation angles based on the third pixel distance, the fourth pixel distance, and the focal length.

[0107] At this point, the third pixel distance can be the pixel distance between the pixel corresponding to the top edge of the bounding box and the center coordinates of the image, and the fourth pixel distance can be the pixel distance between the pixel corresponding to the bottom edge of the bounding box and the center coordinates of the image.

[0108] Furthermore, the vertical rotation angle range can include the angle range between the maximum downward rotation angle of sensor 120 and the maximum upward rotation angle of sensor 120. For example, the vertical rotation angle range can include the angle range between the maximum downward rotation angle of sensor 120 and the maximum upward rotation angle of sensor 120 based on the capture angle of the image captured by sensor 120.

[0109] At this time, the maximum downward rotation angle of sensor 120 may include the rotation angle of sensor 120 based on the position of the upper edge of the user's bounding box at the upper edge of the acquired image by rotating sensor 120 downward. Furthermore, the maximum upward rotation angle of sensor 120 may include the rotation angle of sensor 120 based on the position of the lower edge of the user's bounding box at the lower edge of the acquired image by rotating sensor 120 upward.

[0110] For example, processor 143 can identify the maximum downward rotation angle of sensor 120 as (sensor's vertical FOV) / 2 - tan y = (the sensor ... -1 (Third pixel distance / focal length). Furthermore, the processor 143 can identify the maximum downward rotation angle of the sensor 120 as (sensor's vertical FOV) / 2 + tan y, based on the fact that the y-coordinate value of the pixel corresponding to the upper edge of the bounding box is greater than the y-coordinate value of the center pixel of the image. -1 (Third pixel distance / focal length). In addition, the processor 143 can identify the maximum downward rotation angle of the sensor 120 as (sensor's vertical FOV) / 2, based on the fact that the y-coordinate value of the pixel corresponding to the top edge of the bounding box is the same as the y-coordinate value of the center pixel of the image.

[0111] Furthermore, the processor 143 can identify the maximum upward rotation angle of the sensor 120 as (the sensor's vertical FOV) / 2 + tan y, based on the fact that the y-coordinate value of the pixel corresponding to the lower edge of the bounding box is less than the y-coordinate value of the center pixel of the image. -1(Fourth pixel distance / focal length). Furthermore, the processor 143 can identify the maximum upward rotation angle of the sensor 120 as (sensor's vertical FOV) / 2 - tan y = (sensor's vertical FOV) / 2 - tan y = (fourth pixel distance / focal length). -1 (Fourth pixel distance / focal length). In addition, the processor 143 can identify the maximum upward rotation angle of the sensor 120 as (sensor's vertical FOV) / 2, based on the fact that the y-coordinate value of the pixel corresponding to the lower edge of the bounding box is the same as the y-coordinate value of the image center pixel.

[0112] In addition, the processor 143 can rotate the sensor 120 within a range of rotation angles.

[0113] At this time, the rotation of sensor 120 may include the rotation of sensor 120 and the rotation of sensor 120 according to the rotation of a part of robot 100 in which sensor 120 is disposed.

[0114] For example, if sensor 120 is located within the body of robot 100, processor 143 can control drive component 130 to rotate the body of robot 100 within a rotational angle range. In this case, sensor 120 can rotate along with the body within the rotational angle range. Alternatively, if robot 100 is configured with a first part (e.g., body) and a second part (e.g., head, arm, etc.), and sensor 120 is located at the second part, processor 143 can control drive component 130 to rotate the second part of robot 100 within a rotational angle range. In this case, sensor 120 can rotate along with the second part within the rotational angle range.

[0115] For example, the processor 143 can rotate the sensor 120 to the right and left within the horizontal rotation angle range, and rotate the sensor 120 down and up within the vertical rotation angle range.

[0116] Furthermore, the processor 143 can acquire information using the sensor 120 while the sensor 120 is rotating. In this case, the information may include image and depth information.

[0117] For example, refer to Figure 4 When the sensor 120 is rotating, the sensor 120 can sense the area 410 around the user 10.

[0118] At this time, region 411 can be the region sensed by sensor 120 rotating based on the maximum rotation angle to the left and the maximum rotation angle upward. Furthermore, region 412 can be the region sensed by sensor 120 rotating based on the maximum rotation angle to the right and the maximum rotation angle upward. Furthermore, region 413 can be the region sensed by sensor 120 rotating based on the maximum rotation angle to the left and the maximum rotation angle downward. Furthermore, region 414 can be the region sensed by sensor 120 rotating based on the maximum rotation angle to the right and the maximum rotation angle downward.

[0119] The processor 143 can generate a three-dimensional map of the user's surroundings based on information obtained by sensing the user's surroundings using the sensor 120 (S220).

[0120] For example, processor 143 can obtain a 3D map configured with a point cloud based on image and depth information acquired using sensor 120. According to one or more embodiments, processor 143 can use 3D reconstruction to obtain the 3D map. The point cloud can include a collection of points from 3D space. Each point from the point cloud can include the point's x, y, z coordinate values ​​and RGB values.

[0121] The processor 143 can identify planes from areas of the user's surrounding environment based on a 3D map (S230).

[0122] At this point, the processor 143 can use Random Sample Consensus (RANSAC) to identify the plane from the user's surrounding environment.

[0123] For example, processor 143 can randomly select three points from the points included in a region of a 3D map. Furthermore, processor 143 can identify a plane defined by the selected points. This plane can be represented by an equation of the plane (such as ax + by + cz + d = 0). Additionally, processor 143 can identify the distances between the plane and the remaining points included in the region (excluding the three points), and identify interior points based on the identified distances. Interior points can include points among the points included in the region whose distance from the plane is less than or equal to a threshold.

[0124] The processor 143 can execute the above process multiple times and identify multiple planes and the number of interior points in each of the multiple planes. Furthermore, the processor 143 can identify the plane with the most interior points from among the multiple planes based on the number of interior points in each plane.

[0125] Furthermore, the processor 143 can identify the ratio of the interior points of the identified plane to the points included in the region based on the number of points included in the region and the number of interior points of the identified plane (e.g., the plane with the most interior points). In this case, the processor 143 can identify the region as a plane based on the identification ratio being greater than or equal to a threshold, and identify the region as not a plane based on the identification ratio being less than the threshold.

[0126] Through the above process, the processor 143 can identify areas that are planes from a three-dimensional map of the user's surrounding environment.

[0127] The processor 143 can identify multiple regions on the identified plane that have the same aspect ratio as the projected image (S240).

[0128] At this point, the aspect ratio of the projected image (or projected screen) can include the aspect ratio of the image projected from the projector 110. For example, the aspect ratio can include various ratios, such as 4:3, 5:4, or 16:9.

[0129] For example, processor 143 can identify points included in a plane based on a 3D map and randomly select points from those points. Furthermore, processor 143 can generate candidate regions on the plane by extending a region from the selected point, making the selected point the center. In this case, the region can be extended within the same plane, and the aspect ratio of the extended region can be the same as that of the projected image. Processor 143 can generate multiple candidate regions on the plane by executing the above process multiple times. A candidate region can represent a quadrilateral region on the plane that can be projected onto the image based on the candidate region being a quadrilateral region with the same aspect ratio as the projected image.

[0130] Furthermore, the processor 143 can identify multiple regions from multiple candidate regions based on the location and size of multiple candidate regions.

[0131] For example, processor 143 can determine multiple regions based on the size of multiple candidate regions by selecting multiple regions from the multiple candidate regions in descending order of size. In this case, the number of regions to be selected can be preset.

[0132] At this point, the processor 143 may exclude a candidate region from another candidate region on the plane based on the location of multiple candidate regions. A region being included in another region may mean that the entirety of a region is located within the other region. Therefore, a region may not be considered included in the other region if at least a portion of it is located outside of the other region.

[0133] Processor 143 can identify multiple candidate projection regions from multiple regions (S250).

[0134] For example, processor 143 can identify multiple candidate projection regions from multiple regions based on features of multiple regions. In this case, the features of the regions can include the saturation of the regions. Processor 143 can identify regions with saturation greater than or equal to a threshold based on the RGB values ​​of points included in the multiple regions, and identify the remaining regions in the multiple regions, excluding the identified regions, as multiple candidate projection regions.

[0135] For example, processor 143 can obtain RGB values ​​of points included in multiple regions based on a 3D map, and identify the saturation of multiple regions based on the RGB values. For example, processor 143 can identify the saturation of each point based on the RGB values ​​of the points included in each region, and identify the saturation of each region by calculating the average of the identified saturations. At this time, processor 143 can use the RGB values ​​of all points included in the region to identify the saturation of the region, or select sampling points from all points included in the region and use the RGB values ​​of the selected sampling points to identify the saturation of the region.

[0136] Furthermore, the processor 143 can identify whether there are regions in the multiple regions whose saturation is greater than or equal to a threshold by comparing the saturation of each region in the multiple regions with a threshold. Additionally, based on the regions in the multiple regions whose saturation is greater than or equal to the threshold identified, the processor 143 can determine the remaining regions in the multiple regions, excluding the identified regions, as multiple candidate projection regions.

[0137] Based on the saturation that indicates the brightness of a color, areas with high saturation can indicate that the color of the area is dark. If an image is projected onto a dark area, areas with high saturation can be excluded from the candidate projection areas because the projected image is not easily visible to the user.

[0138] For example, refer to Figure 5a The processor 143 can identify multiple regions 511, 512, 513, 514, 515, 516, 517, and 518 from the user 10's surrounding environment. Each of these regions can be located on a plane, and the aspect ratio of each region can be the same as that of the projected image. Furthermore, refer to... Figure 5b The processor 143 can identify multiple candidate projection regions 512, 513, 515, and 517 from multiple regions 511, 512, 513, 514, 515, 516, 517, and 518. At this time, the saturation of the candidate projection regions can be less than a threshold.

[0139] The processor 143 can identify the priority order of multiple candidate projection regions (S260).

[0140] Priority order can be information that specifies the order in which projection regions can be selected from multiple candidate projection regions. For example, when an image is projected onto candidate projection regions, priority order can include the order in which the projected image is visible to the user among the candidate projection regions.

[0141] For example, processor 143 can identify the priority order of multiple candidate projection regions based on the features of these regions. In this case, the features of the candidate projection regions may include the size of the candidate projection region and the distance between the user and the candidate projection region.

[0142] The processor 143 can identify the size of each candidate projection region among multiple candidate projection regions.

[0143] As described above, a candidate projection region can be generated by extending from a point on the plane, and this point can be the center of the candidate projection region. The processor 143 can identify the size of the candidate projection region based on the extent to which the candidate projection region extends from this point.

[0144] In addition, the processor 143 can identify the distance between the user and each of the multiple candidate projection areas.

[0145] For example, processor 143 can identify the x, y, z coordinate values ​​corresponding to the user's position on the 3D map. As described above, processor 143 can use image and depth information to generate a 3D map of the user's surrounding environment. Processor 143 can obtain the x, y, z coordinate values ​​on the 3D map corresponding to the user's x, y coordinate values ​​during the generation of the 3D map, and identify the x, y, z coordinate values ​​on the 3D map corresponding to the user's position. At this time, the user's x, y coordinate values ​​can include the coordinate values ​​corresponding to the user's position obtained from the image. For example, the user's x, y coordinate values ​​obtained from the image can be the x, y coordinate values ​​of the center pixel of the user's bounding box. If the coordinates of the pixel corresponding to the top left vertex of the bounding box are (x, y, z...) upper_left y upper_left ), and the coordinates of the pixel corresponding to the bottom right vertex of the bounding box are (x lower_right y lower_right Then the coordinates of the center pixel of the bounding box can be represented as ((x) lower_right -x upper_left ) / 2, (y lower_right -y upper_left ) / 2).

[0146] Furthermore, the processor 143 can identify the distance between the user and the candidate projection area by using the x, y, z coordinates of a point corresponding to the center of the candidate projection area and the x, y, z coordinates of the user's position. In this case, the distance can include the Euclidean distance between the two coordinate values.

[0147] The processor 143 can identify the priority order of multiple candidate projection regions based on the size of each of the multiple candidate regions and the distance between the user and each of the multiple candidate projection regions.

[0148] For example, processor 143 can identify the score of a candidate projection region based on the distance between the user and the candidate projection region and the size of the candidate projection region. In this case, the score can be calculated based on Equation 1 below.

[0149] [Equation 1]

[0150] Here, D represents the distance between the user and the candidate projection area, S represents the size of the candidate projection area, and w1 and w2 represent the weight values.

[0151] Furthermore, the processor 143 can identify the priority order of multiple candidate projection regions based on their scores, where candidate projection regions with higher scores have higher priority. Therefore, among the multiple candidate projection regions, those closer to the user and with larger sizes can have a relatively higher priority.

[0152] The processor 143 can display information about the priority order of multiple candidate projection areas at multiple candidate projection areas (S270).

[0153] For example, the processor 143 can control the projector 110 to project an image onto a region that includes multiple candidate projection regions based on the positions and priority order of the multiple candidate projection regions, such that information about the priority order of the multiple candidate projection regions is displayed simultaneously or sequentially on the multiple candidate projection regions.

[0154] To this end, the processor 143 can identify whether multiple candidate projection areas exist within the area where the projector 110 will project the image.

[0155] For example, processor 143 can obtain the three-dimensional coordinates of points corresponding to the vertices of each of the multiple candidate projection regions based on a three-dimensional map. In this case, the three-dimensional coordinates can be coordinates in the world coordinate system.

[0156] Furthermore, the processor 143 can convert three-dimensional coordinate values ​​into two-dimensional coordinate values. In this case, the two-dimensional coordinate values ​​can be coordinates in the pixel coordinate system of the image projected by the projector 110. For example, the processor 143 can use camera calibration as shown in Equation 2 below to convert three-dimensional coordinate values ​​into two-dimensional coordinate values.

[0157] [Equation 2]

[0158] Here, x, y, and z can represent three-dimensional coordinate values, and x and y can represent two-dimensional coordinate values. Furthermore, K can represent the intrinsic parameter matrix, [R|t] can represent the extrinsic parameter matrix, and s can represent the scaling factor.

[0159] The intrinsic parameter matrix may include the intrinsic parameters of the camera (e.g., sensor 120), such as the camera's focal length (e.g., fx, fy) and the camera's principal point (e.g., cx, cy).

[0160] The extrinsic parameter matrix can be a matrix used to transform the world coordinate system to the camera coordinate system, and can include external parameters of the camera, such as the camera's rotation and translation. In this case, the camera's rotation and translation can indicate the camera's attitude. For example, processor 143 can use visual odometry to obtain parameters regarding the camera's rotation and translation to estimate the camera's attitude.

[0161] Furthermore, the processor 143 can identify whether the two-dimensional coordinate values ​​corresponding to the candidate projection area exist within the pixel range of the image projected by the projector 110. In this case, the two-dimensional coordinate values ​​corresponding to the candidate projection area can include the two-dimensional coordinate values ​​converted from the three-dimensional coordinate values ​​of the candidate projection area.

[0162] For example, we can assume that the coordinates of the top-left pixel of the image projected by projector 110 are (0, 0), and the coordinates of the bottom-right pixel of the image are (x, y). width y height Here, x can be determined based on the resolution of the image projected by projector 110. width y height In this case, the pixel range of the image can be 0 ≤ x ≤ x. width , 0≤y≤y height At this point, 0 ≤ x ≤ x width It can be the pixel range of the x-coordinate values, and 0 ≤ y ≤ y height It can be the pixel range of the y-coordinate value.

[0163] The processor 143 can identify that a candidate projection region exists within the area of ​​the image to be projected by the projector 110 based on the fact that both the x-coordinate and y-coordinate values ​​of the two-dimensional coordinates corresponding to the candidate projection region are within the pixel range of the x-coordinate values. Furthermore, the processor 143 can identify that a candidate projection region does not exist within the area of ​​the image to be projected by the projector 110 based on the fact that both the x-coordinate and y-coordinate values ​​of the two-dimensional coordinates corresponding to the candidate projection region are not within the pixel range of the x-coordinate values.

[0164] Using the above method, the processor 143 can identify whether multiple candidate projection areas exist within the area where the projector 110 will project the image.

[0165] The processor 143 can identify, based on the fact that a candidate projection region among multiple candidate projection regions is identified as not existing within the region where the projector 110 will project the image, the rotation angle of the projector 110 that would place the candidate projection region within the region where the projector 110 will project the image, and rotate the projector 110 based on the identified rotation angle. Through the rotation of the projector 110, multiple candidate projection regions can be placed within the region where the projector 110 will project the image.

[0166] For example, processor 143 can identify the rotation angle of projector 110 based on the fact that the x and y coordinate values ​​of the candidate projection area are identified as not existing in the area where the projected image will be projected.

[0167] At this time, the rotation angle of the projector 110 can include the horizontal rotation angle and the vertical rotation angle.

[0168] According to one or more embodiments, the processor 143 can identify x and y coordinate values ​​of the candidate projection region that exist outside the pixel range of the image. For example, the processor 143 can identify x coordinate values ​​of the candidate projection region that exist outside the pixel range of the image's x coordinate values, and identify y coordinate values ​​of the candidate projection region that exist outside the pixel range of the image's y coordinate values.

[0169] Furthermore, the processor 143 can identify each of the identified x-coordinate values ​​based on the recognition of at least one x-coordinate value existing outside the pixel range of the x-coordinate values ​​in the image, relative to the lower limit (e.g., 0) or upper limit (e.g., x) of the pixel range of the x-coordinate value. width The distance between the lower and upper limits. At this point, the processor 143 can identify the distance between the value closest to the x-coordinate value and the x-coordinate value.

[0170] Additionally, the processor 143 can identify each of the identified y-coordinate values ​​based on the recognition of at least one y-coordinate value existing outside the pixel range of the y-coordinate values ​​in the image, by recognizing a lower limit (e.g., 0) or upper limit (e.g., y) of the pixel range of the y-coordinate value. height The distance between the lower and upper limits. At this point, the processor 143 can identify the distance between the value closest to the y-coordinate value and the y-coordinate value in the lower and upper limits.

[0171] Figure 6a , Figure 6b , Figure 6c and Figure 6dThis is an illustration showing an example of multiple regions 620, 630, 640, and 650 defined by two-dimensional coordinate values ​​corresponding to multiple candidate projection regions in the pixel coordinate system of the image 610 projected by the projector 110.

[0172] For example, region 620 could be defined by the first candidate projection region (e.g., Figure 5b The region 630 can be defined by the two-dimensional coordinate values ​​corresponding to the second candidate projection region (e.g., 512). Figure 5b The region 640 can be defined by the corresponding two-dimensional coordinate values ​​of the third candidate projection region (e.g., 513). Figure 5b The region 650 can be defined by the two-dimensional coordinate values ​​corresponding to the fourth candidate projection region (e.g., 515). Figure 5b The area defined by the corresponding two-dimensional coordinate values ​​in 517).

[0173] Reference Figure 6a The processor 143 can identify that the x-coordinate values ​​of pixel 621 corresponding to the upper left vertex of region 620 and pixel 622 corresponding to the lower left vertex of region 620 have exceeded the pixel range of the x-coordinate values ​​of image 610. At this time, the processor 143 can identify the distance between the x-coordinate value of pixel 621 and the lower limit of the pixel range of the x-coordinate values ​​of image 610, and also identify the distance between the x-coordinate value of pixel 622 and the lower limit of the pixel range of the x-coordinate values ​​of image 610.

[0174] Furthermore, the processor 143 can identify the rotation angle of the projector 110 based on the larger of the identified distances. In this case, the rotation angle of the projector 110 may include a leftward rotation angle. For example, information regarding the rotation angle of the projector 110 within the pixel range of the x-coordinate values ​​of the image can be stored in the memory 142 based on the distance between the x-coordinate values ​​and the pixel range of the x-coordinate values. The processor 143 can identify the rotation angle of the projector 110 based on the identified distance and the information regarding the rotation angle of the projector 110.

[0175] In addition, refer to Figure 6b The processor 143 can identify that the y-coordinate values ​​of pixel 631, corresponding to the upper left vertex of region 630, and pixel 632, corresponding to the upper right vertex of region 630, have exceeded the pixel range of the y-coordinate values ​​of image 610. At this time, the processor 143 can identify the distance between the y-coordinate value of pixel 631 and the lower limit of the pixel range of the y-coordinate values ​​of image 610, and also identify the distance between the y-coordinate value of pixel 632 and the lower limit of the pixel range of the y-coordinate values ​​of image 610.

[0176] Furthermore, the processor 143 can identify the rotation angle of the projector 110 based on the larger of the identified distances. In this case, the rotation angle of the projector 110 may include an upward rotation angle. For example, information regarding the rotation angle of the projector 110 within the pixel range of the image's y-coordinate values, based on the distance between the y-coordinate values ​​and the pixel range of the y-coordinate values, can be stored in the memory 142. The processor 143 can identify the rotation angle of the projector 110 based on the identified distance and the information regarding the rotation angle of the projector 110.

[0177] In addition, refer to Figure 6c The processor 143 can identify that the y-coordinate values ​​of pixel 641, corresponding to the lower left vertex of region 640, and pixel 642, corresponding to the lower right vertex of region 640, exceed the pixel range of the y-coordinate values ​​of image 610. At this time, the processor 143 can identify the distance between the y-coordinate value of pixel 641 and the upper limit of the pixel range of the y-coordinate values ​​of image 610, and also identify the distance between the y-coordinate value of pixel 642 and the upper limit of the pixel range of the y-coordinate values ​​of image 610.

[0178] Furthermore, the processor 143 can identify the rotation angle of the projector 110 based on the larger of the identified distances. In this case, the rotation angle of the projector 110 may include a downward rotation angle. For example, information regarding the rotation angle of the projector 110 within the pixel range of the image's y-coordinate values, based on the distance between the y-coordinate values ​​and the pixel range of the y-coordinate values, can be stored in the memory 142. The processor 143 can identify the rotation angle of the projector 110 based on the identified distance and the information regarding the rotation angle of the projector 110.

[0179] In addition, refer to Figure 6d The processor 143 can identify that the x-coordinate values ​​of pixel 651, corresponding to the upper right vertex of region 650, and pixel 652, corresponding to the lower right vertex of region 650, exceed the pixel range of the x-coordinate values ​​of image 610. At this time, the processor 143 can identify the distance between the x-coordinate value of pixel 651 and the upper limit of the pixel range of the x-coordinate values ​​of image 610, and also identify the distance between the x-coordinate value of pixel 652 and the upper limit of the pixel range of the x-coordinate values ​​of image 610.

[0180] Furthermore, the processor 143 can identify the rotation angle of the projector 110 based on the larger of the identified distances. In this case, the rotation angle of the projector 110 can include a rightward rotation angle. For example, information regarding the rotation angle of the projector 110 within the pixel range of the x-coordinate values ​​of the image can be stored in the memory 142 based on the distance between the x-coordinate values ​​and the pixel range of the x-coordinate values. The processor 143 can identify the rotation angle of the projector 110 based on the identified distance and the information regarding the rotation angle of the projector 110.

[0181] As described above, the processor 143 can identify leftward, rightward, upward, or downward rotation angles of the projector 110 that bring at least one candidate projection area located outside the pixel range of the image within the pixel range of the image. At this time, the processor 143 can identify rotation angles of the projector 110 that prevent areas existing within the pixel range of the image from shifting out of the image pixel range due to the rotation of the projector 110. Furthermore, the processor 143 can rotate the projector 110 based on the identified rotation angle.

[0182] In addition, the processor 143 can generate an image that will be projected by the projector 110 based on multiple sub-images.

[0183] According to one or more embodiments, processor 143 can match multiple sub-images with multiple regions of the image projected by projector 110 and generate an image in which multiple sub-images are included in the multiple regions.

[0184] For example, processor 143 can identify multiple regions corresponding to the multiple candidate projection regions from an image to be projected onto a region including the multiple candidate projection regions, based on the positions of the multiple candidate projection regions. In this case, the region corresponding to the candidate projection regions may include a region on the image to be projected defined by two-dimensional coordinate values ​​corresponding to the candidate projection regions.

[0185] In addition, the processor 143 can obtain multiple sub-images corresponding to multiple regions.

[0186] The colors of the multiple sub-images can be different from the color of the remaining area of ​​the image projected by the projector 110. For example, the colors of the multiple sub-images can be a first color, and the color of the remaining area can be a second color that is different from the first color.

[0187] Each of the multiple sub-images may include an indicator corresponding to the priority order of each of the multiple candidate projection regions. The indicator may include numbers or characters indicating the priority order of the candidate projection regions. Characters may include characters that represent numbers in a language or characters with an order (e.g., the alphabet). The indicator may include various graphical elements that can visually provide the user with information about the priority order of the candidate projection regions using numbers or characters, and may be replaced by expressions such as GUIs or icons. The color of the indicator may be different from the color of the background area. In this case, the background area may include the remaining area after excluding the indicators from the sub-images.

[0188] Furthermore, the processor 143 can match multiple sub-images with multiple regions. In this case, the processor 143 can obtain multiple transformation matrices for matching the multiple sub-images with the multiple regions. The transformation matrices may include homography matrices.

[0189] For example, processor 143 can obtain a transformation matrix for matching a sub-image with a region corresponding to the sub-image based on Equation 3 below.

[0190] [Equation 3]

[0191] Here, H represents the homography matrix, (x, y) represents the coordinates of the pixel corresponding to the vertex of the sub-image, and (x′, y′) represents the coordinates of the region corresponding to the sub-image in the image projected by the projector 110.

[0192] Furthermore, the processor 143 can generate an image comprising multiple sub-images in multiple regions by applying multiple transformation matrices to multiple sub-images.

[0193] For example, refer to Figure 7 The processor 143 can apply the transformation matrix H1 to the RGB values ​​of the pixels of the first sub-image 710 corresponding to the first candidate projection region, and match the first sub-image 710 with the region 751 in the image 750 projected by the projector 110 that corresponds to the first candidate projection region. The first candidate projection region can be a candidate projection region with a first priority order, and the first sub-image 710 can include a number 1 711 indicating the priority order of the first candidate projection region.

[0194] Additionally, the processor 143 can apply the transformation matrix H2 to the RGB values ​​of the pixels of the second sub-image 720 corresponding to the second candidate projection region, and match the second sub-image 720 with the region 752 in the image 750 projected by the projector 110 that corresponds to the second candidate projection region. The second candidate projection region can be a candidate projection region with a second priority order, and the second sub-image 720 can include a number 2721 indicating the priority order of the second candidate projection region.

[0195] Additionally, the processor 143 can apply the transformation matrix H3 to the RGB values ​​of the pixels of the third sub-image 730 corresponding to the third candidate projection region, and match the third sub-image 730 with the region 753 in the image 750 to be projected by the projector 110 that corresponds to the third candidate projection region. The third candidate projection region can be a candidate projection region with a third priority order, and the third sub-image 730 can include a number 3731 indicating the priority order of the third candidate projection region.

[0196] Additionally, the processor 143 can apply the transformation matrix H4 to the RGB values ​​of the pixels of the fourth sub-image 740 corresponding to the fourth candidate projection region, and match the fourth sub-image 740 with the region 754 in the image 750 to be projected by the projector 110 that corresponds to the fourth candidate projection region. The fourth candidate projection region can be a candidate projection region with a fourth priority order, and the fourth sub-image 740 can include a number 4741 indicating the priority order of the fourth candidate projection region.

[0197] Furthermore, the processor 143 can control the projector 110 to project an image in which multiple sub-images are included in multiple regions. At this time, when projecting an image, the multiple sub-images included in the multiple regions of the image can be projected onto multiple candidate projection regions respectively.

[0198] For example, refer to Figure 8 The processor 143 can control the projector 110 to project an image 750 including a first sub-image 710, a second sub-image 720, a third sub-image 730, and a fourth sub-image 740.

[0199] At this time, when the projected image 750 is displayed, the first sub-image 710 can be located at the first candidate projection area 810, the second sub-image 720 can be located at the second candidate projection area 820, the third sub-image 730 can be located at the third candidate projection area 830, and the fourth sub-image 740 can be located at the fourth candidate projection area 840.

[0200] According to one or more embodiments, processor 143 may match sub-images corresponding to areas of the projector 110 that will project images, and generate a plurality of images including sub-images for each area.

[0201] For example, processor 143 can, for each candidate projection region, identify a region corresponding to the candidate projection region from the image to be projected onto that region, based on the location of the candidate projection region. In this case, the region corresponding to the candidate projection region may include a region on the image to be projected defined by two-dimensional coordinate values ​​corresponding to the candidate projection region.

[0202] In addition, the processor 143 can obtain multiple sub-images corresponding to multiple regions.

[0203] The colors of the multiple sub-images can be different from the color of the remaining area of ​​the image to be projected by the projector 110. For example, the colors of the multiple sub-images can be a first color, and the color of the remaining area can be a second color that is different from the first color.

[0204] Each of the multiple sub-images may include an indicator corresponding to the priority order of each of the multiple candidate projection regions. The indicator may include numbers or characters indicating the priority order of the candidate projection regions. Characters may include characters that represent numbers in a language or characters with an order (e.g., the alphabet). The indicator may include various graphical elements that can visually provide the user with information about the priority order of the candidate projection regions using numbers or characters, and may be replaced by expressions such as GUIs or icons. The color of the indicator may be different from the color of the background area. In this case, the background area may include the remaining area after excluding the indicators from the sub-images.

[0205] Furthermore, the processor 143 can match sub-images with regions. At this time, the processor 143 can obtain a transformation matrix for matching the sub-image corresponding to the region with the region. Furthermore, the processor 143 can generate an image in which the sub-image is included within the region by applying the transformation matrix to the sub-image.

[0206] For example, refer to Figure 9 The processor 143 can apply the transformation matrix H1 to the RGB values ​​of the pixels of the first sub-image 910 corresponding to the first candidate projection region, and match the first sub-image 910 with the region 951 in the image 950 projected by the projector 110 that corresponds to the first candidate projection region. The first candidate projection region can be a candidate projection region with a first priority order, and the first sub-image 910 can include a number 1 911 indicating the priority order of the first candidate projection region.

[0207] Additionally, the processor 143 can apply the transformation matrix H2 to the RGB values ​​of the pixels of the second sub-image 920 corresponding to the second candidate projection region, and match the second sub-image 920 with the region 961 in the image 960 projected by the projector 110 that corresponds to the second candidate projection region. The second candidate projection region can be a candidate projection region with a second priority order, and the second sub-image 920 can include a number 2921 indicating the priority order of the second candidate projection region.

[0208] Additionally, the processor 143 can apply the transformation matrix H3 to the RGB values ​​of the pixels of the third sub-image 930 corresponding to the third candidate projection region, and match the third sub-image 930 with the region 971 in the image 970 to be projected by the projector 110 that corresponds to the third candidate projection region. The third candidate projection region can be a candidate projection region with a third priority order, and the third sub-image 930 can include a number 3931 indicating the priority order of the third candidate projection region.

[0209] Additionally, the processor 143 can apply the transformation matrix H4 to the RGB values ​​of the pixels of the fourth sub-image 940 corresponding to the fourth candidate projection region, and match the fourth sub-image 940 with a region 981 in the image 980 projected by the projector 110 that corresponds to the fourth candidate projection region. The fourth candidate projection region can be a candidate projection region with a fourth priority order, and the fourth sub-image 940 can include a number 4941 indicating the priority order of the fourth candidate projection region.

[0210] Furthermore, the processor 143 can control the projector 110 to sequentially project multiple images, each sub-image being included in each region. At this time, when projecting an image, the sub-images included in the regions of the image can be projected at each candidate projection region. The processor 143 can control the projector 110 to begin projection from the image including the sub-image corresponding to the candidate projection region with the highest priority, based on the priority order of the multiple candidate projection regions.

[0211] For example, refer to Figure 10a The processor 143 can control the projector 110 to project an image 950 including the first sub-image 910. At this time, when the image 950 is projected, the first sub-image 910 can be located at the first candidate projection area 1010.

[0212] Reference Figure 10b The processor 143 can control the projector 110 to project an image 960 including the second sub-image 920. At this time, when the image 960 is projected, the second sub-image 920 can be located at the second candidate projection area 1020.

[0213] Reference Figure 10c The processor 143 can control the projector 110 to project an image 970 including the third sub-image 930. At this time, when the image 970 is projected, the third sub-image 930 can be located at the third candidate projection area 1030.

[0214] Reference Figure 10d The processor 143 can control the projector 110 to project an image 980 including the fourth sub-image 940. At this time, when the image 980 is projected, the fourth sub-image 940 can be located at the fourth candidate projection area 1040.

[0215] The processor 143 can identify the candidate projection region selected from multiple candidate projection regions based on user input as the projection region (S280).

[0216] User input can be entered in various ways.

[0217] According to one or more embodiments, processor 143 may receive a speech signal for a user's speech. For example, processor 143 may convert the received speech signal for a user's speech into text, and recognize indicators included in the user's speech based on the text. Processor 143 may identify a candidate projection region having the recognized indicators as a projection region from a plurality of candidate projection regions.

[0218] According to one or more embodiments, processor 143 can use gesture recognition to receive user input. For example, processor 143 can use sensor 120 to obtain an image of the user captured, and analyze the image to obtain the three-dimensional coordinate values ​​of a first point and a second point of the user's body parts. In this case, the first point may be the user's elbow, and the second point may be the user's wrist.

[0219] Furthermore, the processor 143 can identify vectors corresponding to the user's body parts based on the three-dimensional coordinate values ​​of the first and second points. For example, the processor 143 can obtain a vector based on the three-dimensional coordinate values ​​of the first and second points. In this case, the starting point of the vector can be the three-dimensional coordinate value of the first point, and the ending point of the vector can be the three-dimensional coordinate value of the second point. Furthermore, the processor 143 can identify candidate projection regions existing in the direction the vector faces as projection regions based on the vector and the three-dimensional coordinate values ​​of multiple candidate projection regions. For example, the processor 143 can identify the points where the vector intersects with planes in the user's surrounding environment, and identify the distance between the identified points and the center point of each of the multiple candidate projection regions. The processor 143 can identify the candidate projection region with the shortest distance among the multiple candidate projection regions as the candidate projection region existing in the direction the vector faces based on the identified distances.

[0220] The processor 143 can use the projector 110 to project image content onto the projection area (S290).

[0221] The projection area can include the area that provides image content to the user.

[0222] For example, processor 143 can control projector 110 to project image content corresponding to user input onto the projection area.

[0223] According to one or more embodiments, processor 143 can convert speech signals into text based on received speech signals for user speech, and obtain information about user intent by analyzing the meaning of the text. Processor 143 can obtain image content corresponding to the user intent and control projector 110 to project the image content onto the projection area.

[0224] For example, if a user's intent is related to a search (or request) for image content, the processor 143 can obtain keywords by analyzing text and search for image content associated with those keywords. In this case, the processor 143 can search for image content associated with the keywords from multiple image contents stored in the memory 142. Alternatively, the processor 143 can obtain image content associated with the keywords from a server providing streaming services. Furthermore, the processor 143 can obtain image content associated with the keywords by requesting a search for those keywords from websites, search engines, etc.

[0225] For example, image content can include various images such as television programs, movies, and dramas. Additionally, image content can include information such as weather and time, or information related to services provided within the space where robot 100 is located, as well as advertising images.

[0226] As described above, the robot 100 can receive user input through interaction with the user and project image content corresponding to the user input around the user. In this disclosure, the user can be described as an object of interaction.

[0227] For example, processor 143 can use a transformation matrix to match the image content with the region corresponding to the projection area in the image projected by projector 110, and control projector 110 to project an image including the image content. At this time, when projecting an image, the image content included in the image can be projected onto the projection area.

[0228] For example, a first candidate projection region can be assumed among multiple candidate projection regions (e.g., Figure 5b512) has been identified as the projection region. Processor 143 can use the transformation matrix H1 and match it to the region in the image to be projected by projector 110 that corresponds to the first candidate projection region. Furthermore, referring to... Figure 11 The processor 143 can control the projector 110 to project an image 1110 including image content 1111. At this time, when the image 1110 is projected, the image content 1111 can be located at the first candidate projection area 1120.

[0229] Furthermore, in the above example, when information about the priority order of multiple candidate projection areas is displayed sequentially in multiple candidate projection areas, it has been described that the projection includes images of indicators such as numbers or characters, but is not limited to the above.

[0230] The processor 143 can control the projector 110 to project images in sequence according to user input.

[0231] At this point, the image may include a sub-image that will be projected onto the candidate projection area. The sub-image may be monochrome and may not include indicators corresponding to the priority order.

[0232] For example, processor 143 can identify the candidate projection region with the highest priority based on the priority order of multiple candidate projection regions, and control projector 110 to project an image corresponding to the identified candidate projection region. For example, refer to Figure 12a The processor 143 can control the projector 110 to project an image 1210 including the sub-image 1211. At this time, when the image 1210 is projected, the sub-image 1211 can be located at the first candidate projection area 1220.

[0233] The processor 143 can receive user input while the image is being projected, and control the projector 110 to project images corresponding to candidate projection regions with the second highest priority order based on the user input, or identify the candidate projection regions of the currently projected sub-image as projection regions.

[0234] For example, processor 143 can convert a speech signal into text based on the received speech signal for the user's speech, and obtain information about the user's intent by analyzing the meaning of the text. Processor 143 can control projector 110 to project images corresponding to candidate projection regions with the second highest priority order based on the user's intent, or identify the candidate projection region of the currently projected sub-image as the projection region.

[0235] For example, refer to Figure 12bThe user can utter a voice command such as "next" 1230 while image 1210 is being projected. Processor 143 can receive the voice signal and recognize the user's intention by converting the voice signal into text. Processor 143 can recognize the user's voice command such as "next" 1230 as including an intention to request projection of an image corresponding to a candidate projection region with the second-highest priority. In this case, processor 143 can identify a candidate projection region with the second-highest priority order after the candidate projection region of the currently projected sub-image based on the priority order of multiple candidate projection regions, and control projector 110 to project an image corresponding to the identified candidate projection region. For example, refer to... Figure 12c The processor 143 can control the projector 110 to project an image 1240 including the sub-image 1241. At this time, when the image 1240 is projected, the sub-image 1241 can be located at the second candidate projection area 1250.

[0236] For example, such as Figure 12d As shown, a user can utter a voice such as "here" 1260 while image 1210 is being projected. Processor 143 can receive the voice signal and recognize the user's intention by converting the voice signal into text. Processor 143 can recognize user voice such as "here" 1260 as including the intention to select a candidate projection region of the currently projected sub-image as the projection region. In this case, processor 143 can recognize the first candidate projection region 1220 of the projected sub-image 1211 as the projection region.

[0237] As described above, the processor 143 can project images corresponding to candidate projection regions one by one based on user input, and determine candidate projection regions from multiple candidate projection regions.

[0238] As described above, robot 100 can identify the projection area from the user's surrounding environment.

[0239] For example, robot 100 can acquire information using sensor 120 (e.g., a LiDAR sensor, a camera, etc.) and explore its surroundings using the acquired information to identify its environment. Identifying its environment may include acquiring information about the orientation of objects within its environment, the distance between the robot and the objects, etc. Additionally, robot 100 can detect and track users from images acquired using sensor 120. For example, robot 100 can track users by comparing their feature information (e.g., size, color, shape, contour, etc.) between image frames and assign an ID to each user for differentiation.

[0240] At this time, the processor 143 can identify the user who made the voice signal based on the received voice signal including the trigger word, and identify the projection area from the environment around the identified user.

[0241] Trigger words can include words used to call robot 100. Trigger words can be preset words or words selected from multiple preset words by user input.

[0242] For example, processor 143 can convert the speech signal into text based on the received speech signal for the user's speech, and identify whether the trigger word is included in the user's speech by analyzing the text.

[0243] Furthermore, the processor 143 can identify the user who made the speech containing the trigger word based on the trigger word being recognized as being included in the user's speech, and identify the projection area from the environment surrounding the identified user.

[0244] According to one or more embodiments, robot 100 may include multiple microphones. Processor 143 may identify the user uttering the voice based on the fact that voice signals are received through the multiple microphones by recognizing the direction from which the user's voice is received. For example, processor 143 may use sound localization to identify the direction of the sound source and identify the user located in the identified direction as the user uttering the trigger word. Furthermore, processor 143 may use an image including the user uttering the trigger word to identify a projection area from the user's surrounding environment. In this case, processor 143 may control drive component 130 to rotate robot 100 so that the field of view of robot 100 (e.g., the field of view of sensor 120) faces the user uttering the trigger word, and use sensor 120 to obtain an image including the user uttering the trigger word.

[0245] According to one or more embodiments, processor 143 may identify the user who issued the trigger word based on an image obtained by using sensor 120.

[0246] For example, processor 143 can detect the contour of an object from an image and calculate a probability value indicating the degree to which the shape of a pre-stored object (e.g., a mouth) matches the detected contour. Furthermore, processor 143 can identify the object with the highest probability value among the probability values ​​calculated for the object's contour as the user's mouth. In this case, processor 143 can identify the user whose mouth shape changes based on the recognition that a speech signal including a trigger word has been received, from an image frame obtained from the time point of receiving the speech signal (or the time period from a preset time before the time point of receiving the speech signal to the time point of receiving the speech signal), and identify the identified user as the user who uttered the trigger word. Additionally, processor 143 can use an image including the user who uttered the trigger word to identify a projection area from the user's surrounding environment.

[0247] Figure 13 This is a block diagram illustrating the configuration of a robot according to one or more embodiments.

[0248] Reference Figure 13 The robot 100 may include a projector 110, a sensor 120, a drive unit 130, a main module 140, an input interface 150, and an output interface 160. Furthermore, the above configuration is an example, and new configurations can be added. Additionally, the detailed configuration can be compared with... Figure 1a The configurations shown overlap, therefore, for Figure 13 Additional implementation details of one or more embodiments shown can be found in [reference]. Figure 1a And the corresponding description.

[0249] Sensor 120 can detect the structure of a space or objects. Objects can include walls and obstacles in the space. Obstacles can include various objects present in the space, such as, but not limited to, furniture, home appliances, remote controls, keys, people, pets, etc. Furthermore, the information obtained by sensor 120 can be used to generate a map of the space.

[0250] Sensor 120 may include a LiDAR sensor, an obstacle detection sensor, and a motion detection sensor. The LiDAR sensor can output laser light in a 360-degree direction. When it receives reflected laser light from an object, it analyzes the time difference between the laser's reflection and return, the signal strength of the received laser light, and other factors to obtain spatial geometric information. This geometric information may include the object's position, distance, and orientation. The LiDAR sensor can then provide this geometric information to processor 143.

[0251] Obstacle detection sensors can detect obstacles around the robot. For example, obstacle detection sensors include at least one of ultrasonic sensors, infrared sensors, radio frequency (RF) sensors, geomagnetic sensors, and position-sensitive device (PSD) sensors. The obstacle detection sensors can detect obstacles present on the robot's front, rear, side surfaces, or path of movement. The obstacle detection sensors can provide the detected obstacle information to processor 143.

[0252] The motion detection sensor can detect the movement of robot 100. For example, the motion detection sensor may include at least one of a gyroscope sensor, a wheel encoder, or an accelerometer. The gyroscope sensor can detect the rotation direction and rotation angle of robot 100. The wheel encoder can detect the number of revolutions of the wheels of robot 100. The accelerometer can detect the speed change of robot 100. The motion detection sensor can provide the detected motion information to processor 143.

[0253] For example, processor 143 can use information obtained through sensor 120 to generate a map of the space. The map can be generated during the initial exploration of the space. For instance, processor 143 can explore the space using a LiDAR sensor to obtain geometric information about the space and use that geometric information to generate a map of the space. In this case, the map may include a grid map.

[0254] In addition, the processor 143 can use simultaneous localization and mapping (SLAM) to identify the location of the robot 100 on the map.

[0255] For example, processor 143 can use a LiDAR sensor to obtain spatial geometry information and identify the position of robot 100 on a map by comparing the obtained geometry information with pre-stored geometry information, or by comparing the obtained geometry information. However, this disclosure is not limited to the above example, and for example, processor 143 can identify the position of robot 100 on a map by using camera-based SLAM.

[0256] In addition, the processor 143 can use the information obtained through the sensor 120 to control the movement of the robot 100.

[0257] For example, processor 143 can use a map stored in memory 142 to control drive unit 130 to make robot 100 move in space. Furthermore, processor 143 can use sensor 120 to acquire information while robot 100 is moving in space, and use the acquired information to detect obstacles around robot 100. When an obstacle is detected, processor 143 can determine the movement pattern of robot 100 (e.g., linear movement and rotation), and control drive unit 130 according to the determined movement pattern to make robot 100 avoid the obstacle. Additionally, processor 143 can use information acquired by sensor 120 to identify movement information such as robot 100's moving speed and the distance robot 100 has traveled, and update robot 100's position on the map based on the movement information.

[0258] Input interface 150 may include circuitry. Input interface 150 may receive user input and transmit the user input to processor 143. For example, input interface 150 may receive various user inputs for setting or selecting various functions supported by robot 100.

[0259] Input interface 150 may include various types of input devices.

[0260] According to one or more embodiments, input interface 150 may include physical buttons. Physical buttons may include function keys or dial buttons. Physical buttons may be implemented as one or more keys.

[0261] According to one or more embodiments, the input interface 150 may use a touch method to receive user input. For example, the input interface 150 may be implemented using a touchscreen capable of performing the functions of the display 161.

[0262] According to one or more embodiments, the input interface 150 may use a microphone to receive voice signals in response to a user's voice. In this case, the input interface 150 may include one or more microphones. The processor 143 may use the voice signals to perform functions corresponding to the user's voice.

[0263] The output interface 160 may include a display 161 and a speaker 162.

[0264] The display 161 can display various screens. The processor 143 can display various notifications, messages, information, etc., associated with the operation of the robot 100 on the display 161.

[0265] Display 161 can be implemented as a display including a self-emissive device or a display including a non-emissive device and a backlight. For example, display 161 can be implemented as various forms of display, such as, but not limited to, liquid crystal display (LCD), organic light-emitting diode (OLED) display, light-emitting diode (LED) display, micro LED display, small LED display, quantum dot light-emitting diode (QLED) display, etc.

[0266] Speaker 162 can output audio signals. Processor 143 can output warning sounds, notification messages, and response messages corresponding to user input, etc., associated with the operation of robot 100 through speaker 162.

[0267] Figure 14 This is a flowchart illustrating a method for projecting images through a robot according to one or more embodiments.

[0268] The robot can identify multiple candidate projection regions based on information obtained by sensing the user's surrounding environment (S1410).

[0269] The robot can identify the priority order of multiple candidate projection regions (S1420).

[0270] The robot can control the projector to project an image onto a region that includes multiple candidate projection regions based on the positions and priority order of the multiple candidate projection regions, and simultaneously or sequentially display information about the priority order of the multiple candidate projection regions at the multiple candidate projection regions (S1430).

[0271] The robot can identify the candidate projection region selected from multiple candidate projection regions based on user input as the projection region (S1440).

[0272] The robot can use a projector to project image content onto the projection area (S1450).

[0273] In operation S1430, the robot can identify multiple regions corresponding to the multiple candidate projection regions of the image to be projected onto the region, based on the positions of the multiple candidate projection regions, and control the projector to project multiple sub-images included in the multiple regions. Each of the multiple sub-images may include an indicator corresponding to the priority order of each of the multiple candidate projection regions. The indicator may include a number indicating the priority order of the candidate projection regions.

[0274] In operation S1430, the robot can, for each of the candidate projection regions, identify the region of the image to be projected onto that region based on the position of the candidate projection region, and control the projector to sequentially project multiple images included in each region for each sub-image. Each sub-image may include an indicator corresponding to the priority order of each candidate projection region among the multiple candidate projection regions. The indicator may include a number indicating the priority order of the candidate projection regions.

[0275] In operation S1420, the robot can identify the priority order of multiple candidate projection regions based on the size of each of the multiple candidate projection regions and the distance between the user and each of the multiple candidate projection regions.

[0276] In operation S1410, the robot can generate a three-dimensional map of the user's surrounding environment based on information obtained by sensing the user's surrounding environment through sensors, identify planes in the user's surrounding environment based on the three-dimensional map, identify multiple regions on the identified planes that have the same aspect ratio as the projected image, and identify multiple candidate projection regions from the multiple regions based on the features of the multiple regions.

[0277] In operation S1410, the robot can identify regions with saturation greater than or equal to a threshold in multiple regions based on the RGB values ​​of points included in the multiple regions, and identify the remaining regions in the multiple regions other than the identified regions as multiple candidate projection regions.

[0278] In operation S1410, the robot can use sensors to acquire images, identify the user's position and the sensor's field of view based on the images, and acquire information using the sensors while the sensors rotate within the rotation angle range.

[0279] In this scenario, the robot can obtain the bounding box for the user from the acquired image, identify the pixel distance between the center pixel of the acquired image and the pixels of the bounding box, and identify the range of the sensor's rotation angle based on the sensor's focal length and pixel distance.

[0280] Furthermore, according to one or more embodiments, the various embodiments described above can be implemented using software including instructions stored in a machine-readable storage medium (e.g., a computer). A machine can invoke the instructions stored in the storage medium, and as a means operable according to the invoked instructions, it may include an electronic device according to the embodiments described above. Based on the command executed by the processor, the processor can perform the function corresponding to the command directly or using other elements under the processor's control. The command may include code generated by a compiler or executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is tangible and does not include signals, and this term does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0281] Additionally, according to one or more embodiments, methods according to the various embodiments described above can be provided within a computer program product. The computer program product can be traded as a commodity between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or published online through an app store (e.g., PLAYSTORE™). In the case of online publication, at least a portion of the computer program product can be stored, at least temporarily, in a storage medium such as the memory of a manufacturer's server, an app store's server, or a relay server, or can be temporarily generated.

[0282] Additionally, according to one or more embodiments, the various embodiments described above can be implemented in a recordable medium readable by a computer or computer-like device using software, hardware, or a combination thereof. The embodiments described herein can be implemented by the processor itself. According to software implementations, embodiments such as the processes and functions described herein can be implemented using separate software modules. Each software module can perform one or more of the functions and operations described herein.

[0283] Furthermore, computer instructions for performing processing operations in the apparatus according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When executed by the processor of the apparatus, the computer instructions stored in the non-transitory computer-readable medium can cause the apparatus to perform the processing operations in the apparatus according to the various embodiments described above. A non-transitory computer-readable medium can refer to a medium that stores data semi-permanently rather than for a very short period of time, such as registers, caches, memories, etc., and can be read by the apparatus. Examples of non-transitory computer-readable media may include, but are not limited to, disc compacts (CDs), digital versatile discs (DVDs), hard disks, Blu-ray discs, USB, memory cards, ROMs, etc.

[0284] Furthermore, each of the elements (e.g., modules or programs) according to the various embodiments described above may be formed by a single entity or multiple entities, and other sub-elements may be further included in the various embodiments. A portion of an element (e.g., a module or program) may be integrated into a single entity to perform the same or similar functions performed by each related element prior to integration. According to the various embodiments, the operations performed by a module, program, or other element may be performed sequentially, in parallel, repeatedly, or heuristically, or at least some of the operations may be performed in a different order, or different operations may be added.

[0285] While this disclosure has been shown and described with reference to various exemplary embodiments, it will be understood that these exemplary embodiments are intended to be illustrative and not restrictive. Those skilled in the art will understand that various changes in form and detail may be made therein without departing from the true spirit and full scope of this disclosure, including the appended claims and their equivalents.

Claims

1. A robot, comprising: Projector; sensor; Memory, storing instructions; as well as One or more processors, The instructions, when executed jointly or individually by the one or more processors, cause the robot to: Multiple candidate projection areas are identified based on first information obtained by sensing the user's surrounding environment via the sensor; Identify the priority order of the multiple candidate projection regions; Control the projector: Based on the multiple positions of the multiple candidate projection regions and the priority order, the first image is projected onto the region including the multiple candidate projection regions. Second information regarding the priority order is displayed in the plurality of candidate projection regions; The candidate projection regions selected from the plurality of candidate projection regions based on user input are identified as projection regions; and The image content is projected onto the projection area via the projector.

2. The robot as described in claim 1, wherein, The one or more processors are configured to execute the instructions to cause the robot to: Based on the multiple locations, multiple regions of the second image to be projected into the region are identified, and the multiple regions correspond to the multiple candidate projection regions; as well as Control the projector to project a second image, and The second image includes multiple sub-images corresponding to the multiple regions, and the multiple sub-images include multiple indicators corresponding to the priority order.

3. The robot according to claim 2, wherein, The plurality of indicators includes a plurality of numbers indicating the priority order.

4. The robot as claimed in claim 1, wherein, The one or more processors are configured to execute the instructions to cause the robot to: Based on the multiple locations, identify multiple regions corresponding to the multiple candidate projection regions; as well as Control the projector to sequentially project multiple sub-images corresponding to the multiple regions, and The plurality of sub-images include a plurality of indicators corresponding to the priority order.

5. The robot according to claim 4, wherein, The plurality of indicators includes a plurality of numbers indicating the priority order.

6. The robot as claimed in claim 1, wherein, The one or more processors are configured to execute the instructions so that the robot identifies the priority order based on: The multiple dimensions of the multiple candidate projection regions, and The multiple distances between the user and the multiple candidate projection areas.

7. The robot as claimed in claim 1, wherein, The one or more processors are configured to execute the instructions to cause the robot to: A three-dimensional map of the surrounding environment is generated based on the first information; Identify planes from the surrounding environment based on the 3D map; Identify multiple regions on the plane having a first aspect ratio, the first aspect ratio being matched with a second aspect ratio of the projected image; as well as The multiple candidate projection regions are identified from the multiple regions based on the features of the multiple regions.

8. The robot according to claim 7, wherein, The one or more processors are configured to execute the instructions to cause the robot to identify the remaining areas in the plurality of regions, excluding the identified areas, as the plurality of candidate projection regions. Wherein, the saturation of the identified region is greater than or equal to a threshold, and The identified regions are determined based on the RGB values ​​of multiple points within the multiple regions.

9. The robot as claimed in claim 1, wherein, The one or more processors are configured to execute the instructions to cause the robot to: A third image is obtained via the sensor; Identify the user's position in the third image; The rotation angle range of the sensor is identified based on the user's position and the sensor's field of view; as well as While the sensor rotates within the rotation angle range, the sensor obtains the first information.

10. The robot of claim 9, wherein, The one or more processors are configured to execute the instructions to cause the robot to: Obtain the bounding box for the user based on the third image; Identify the pixel distance between the center pixel of the third image and the pixels of the bounding box; as well as The rotation angle range is identified based on the focal length of the sensor and the pixel distance.

11. A method for projecting images via a robot including a projector, the method comprising: Multiple candidate projection areas are identified based on first information obtained by sensing the user's surrounding environment via sensors; Identify the priority order of the multiple candidate projection regions; Based on the multiple positions of the multiple candidate projection regions and the priority order, the first image is projected onto the region including the multiple candidate projection regions via the projector. Second information regarding the priority order is displayed in the plurality of candidate projection regions; The candidate projection region selected from the plurality of candidate projection regions based on user input is identified as the projection region; as well as The image content is projected onto the projection area via the projector.

12. The method according to claim 11, wherein, The display includes: Based on the multiple locations, multiple regions of a second image to be projected into the region are identified, the multiple regions corresponding to the multiple candidate projection regions; and Control the projector to project a second image, and The second image includes multiple sub-images corresponding to the multiple regions, and the multiple sub-images include multiple indicators corresponding to the priority order.

13. The method according to claim 12, wherein, The plurality of indicators includes a plurality of numbers indicating the priority order.

14. The method according to claim 11, wherein, The display steps include: Based on the multiple locations, identify multiple regions corresponding to the multiple candidate projection regions; and Control the projector to sequentially project multiple sub-images corresponding to the multiple regions, and The plurality of sub-images include a plurality of indicators corresponding to the priority order.

15. The method according to claim 14, wherein, The plurality of indicators includes a plurality of numbers indicating the priority order.

Images