Customized service providing device and method

The customized service provision device uses visual sensors and analysis models to actively recognize user situations, addressing the limitations of current robots by providing accurate and proactive services through integrated sensing data and adaptive judgment, enhancing interaction robots' usability and accuracy.

WO2024232490A1PCT designated stage expired Publication Date: 2024-11-14ROBOCARE CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2023/018410
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-08
Filing Date
2023-11-16
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Current interaction robots struggle to accurately recognize the user's situation and provide appropriate services, relying mainly on passive responses to user input, which limits their ability to actively assist, especially in complex environments like those of elderly individuals.

Method used

A customized service provision device and method that utilizes a combination of visual sensors, analysis models, and various sensing data to actively recognize the user's situation, integrating element information to generate situation information and provide tailored services, incorporating both rule-based and machine learning models for enhanced accuracy.

Benefits of technology

Enables accurate and proactive service provision tailored to the user's situation, reducing misrecognition and enhancing the robot's usability by leveraging diverse sensing data and adaptive judgment criteria.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2023018410_14112024_PF_FP_ABST
    Figure KR2023018410_14112024_PF_FP_ABST
Patent Text Reader

Abstract

A customized service providing device according to an embodiment of the present invention comprises: a moving unit controlled to move to a place where a user can be recognized; a collection unit which collects image capture information related to the user through a visual sensor; a deriving unit which inputs the information collected through the collection unit into at least one analysis model and derives element information for each of the analysis models; a situation recognition unit which integrates the derived element information for each of the analysis models to generate integrated information, and recognizes the current situation of the user on the basis of the integrated information to generate situation information; and a service providing unit which provides the user with a customized service on the basis of the situation information.
Need to check novelty before this filing date? Find Prior Art

Description

Device and method for providing customized services

[0001] The present invention relates to a device, method and computer program for providing customized services using a robot.

[0002] Recent robot development is being reoriented towards the development of companion robots that are closely related to and can deeply penetrate human lives, rather than simply mechanical computing devices as in the past. These companion robots are called interaction robots because they interact with humans.

[0003] However, most of the interactions performed by currently released interactive robots, such as smart speakers and toy robots, are simple and passive actions that receive input from users and output responses accordingly. Therefore, it is difficult for robots to recognize the user's situation and actively provide the necessary services to the user according to the situation. Since it is more appropriate to view them as simple computing devices, the differentiation from them is becoming more important.

[0004] Especially for older adults who spend a significant amount of time at home, aging often leads to health problems and limitations in daily life. In these situations, family members or caregivers provide care, but maintaining constant care in all circumstances requires significant time and expense.

[0005] However, as mentioned above, the currently developed robots have a problem in accurately recognizing the situation of the user who is the object of care.

[0006] Currently developed care robots offer voice guidance, emergency assistance, and media playback services. However, these services are often provided only after the user has requested a specific service. Consequently, it is difficult for robots to actively recognize the user's situation and provide appropriate services accordingly.

[0007] [Prior Art Document] (Patent Document 1) Korean Patent Publication No. 10-2004-0034164 (Registered on February 26, 2023)

[0008] The present invention aims to provide a customized service provision device and method that actively and accurately recognizes the user's situation using a robot and provides a service corresponding to the user's situation.

[0009] In addition, the present invention seeks to provide a customized service providing device and method that utilizes not only visual information but also various types of sensing data to enable a robot to more accurately recognize a user's situation and thereby increase the user's usability of the robot.

[0010] In addition, the present invention aims to provide a customized service providing device and method for reducing the probability of misrecognizing a user's situation by judging the user's situation based on collected information and making judgments based on different judgment criteria depending on the status and type of the collected information.

[0011] However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

[0012] As a means for achieving the above-described technical task, one embodiment of the present invention comprises a device for providing a user-customized service, the device comprising: a moving unit controlled to move to a location where the user can be recognized; a collecting unit for collecting photographic information related to the user through a visual sensor; a deriving unit for inputting information collected through the collecting unit into at least one analysis model and deriving element information for each analysis model; a situation recognition unit for recognizing the current situation of the user based on the derived element information and generating situation information; and a service providing unit for providing a customized service to the user based on the situation information.

[0013] In addition, another embodiment of the present invention provides a method for providing a user-customized service using a robot, comprising: a moving step of controlling a mobile device to move to a location where the user can be recognized; a collecting step of collecting photographic information related to the user through a visual sensor; an analyzing step of inputting the information collected in the collecting step into at least one analysis model to derive element information for each analysis model; a situation recognition step of generating integrated information by integrating the derived element information for each analysis model and recognizing the current situation of the user based on the integrated information to generate situation information; and a service providing step of providing a customized service to the user based on the situation information.

[0014] The above-described problem-solving methods are merely exemplary and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, additional embodiments may exist, as described in the drawings and detailed description of the invention.

[0015] According to any one of the above-described problem solving means of the present invention, a robot can be used to actively and accurately recognize a user's situation and provide a service corresponding to the user's situation.

[0016] In addition, the present invention can increase the user's usability of the robot by utilizing various types of sensing data as well as visual information, thereby allowing the robot to more accurately recognize the user's situation.

[0017] In addition, the present invention can reduce the probability of incorrectly recognizing the user's situation by judging the user's situation based on collected information by making different judgment criteria according to the status and type of the collected information.

[0018] Figure 1 is a configuration diagram of a customized service provision system according to one embodiment of the present invention.

[0019] Figure 2 is a configuration diagram of a customized service providing device according to one embodiment of the present invention.

[0020] FIGS. 3 to 10, 11a and 11b are diagrams illustrating analysis models for analyzing shooting information in a customized service providing device according to one embodiment of the present invention.

[0021] FIG. 12 is a drawing illustrating a process of recognizing a situation by a customized service providing device according to one embodiment of the present invention.

[0022] FIG. 13 and FIG. 14 are drawings for explaining that a customized service providing device according to one embodiment of the present invention provides a customized service.

[0023] FIG. 15 and FIG. 16 are diagrams illustrating a situation recognition unit according to an embodiment of the present invention generating situation information according to a rule-based model.

[0024] FIG. 17 and FIG. 18 are diagrams illustrating how a situation recognition unit according to an embodiment of the present invention generates situation information according to a machine learning model.

[0025] FIG. 19 is a diagram illustrating a situation recognition unit according to an embodiment of the present invention that generates situation information by utilizing a rule-based model and a machine learning model together.

[0026] FIG. 20 is a drawing for explaining that a customized service providing device according to one embodiment of the present invention provides a customized service.

[0027] Figure 21 is a flowchart illustrating a method for providing customized services according to one embodiment of the present invention.

[0028] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity of description, and similar reference numerals have been used throughout the specification to indicate similar elements.

[0029] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Furthermore, when a part is said to "include" a component, this should be understood to mean that, unless specifically stated to the contrary, it may include other components rather than excluding them, and does not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] In this specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Furthermore, a single unit may be realized using two or more pieces of hardware, and two or more units may be realized by a single piece of hardware.

[0031] Some of the operations or functions described herein as being performed by a terminal or device may instead be performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by a server may also be performed by a terminal or device connected to the server.

[0032] Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0033] Figure 1 is a configuration diagram of a customized service provision system according to one embodiment of the present invention.

[0034] Referring to FIG. 1, a customized service providing system (1) may include a customized service providing device (100).

[0035] Each component of the customized service providing device (100) of FIG. 1 is generally connected via a network. For example, as illustrated in FIG. 1, the customized service providing device (100) and the server (200) may be connected to the network simultaneously or at time intervals. In addition, the network refers to a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW), wired and wireless data communication networks, telephone networks, wired and wireless television communication networks, etc. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth, infrared, ultrasonic, visible light communication (VLC), and LiFi.

[0036] The customized service providing device (100) operates based on a built-in power source or an external power source and may be exemplified as a robot for communicating or interacting with a user (10). More specifically, the customized service providing device (100) may include a body part (110), a moving part (120), and may further include additional components necessary for driving other robots.

[0037] The body (110) is a central configuration in which major components of the customized service providing device (100), such as a CPU (not shown), a battery (not shown), a communication unit (117), etc., are mounted internally or other components are connected externally, and may further include a collection unit (111) for capturing a user or sensing the user's voice or ambient sound to generate related information, a display unit (115) for outputting images or text, and a speaker unit (116) for outputting voice.

[0038] The collection unit (111) can capture images of the user (10) or the environment around the user through a visual sensor such as a camera. Here, the components included in the visual sensor may include, for example, photographing devices such as a camera that generates photographing information of general RGB pixels, an infrared camera for capturing an object in a dark environment, and a depth camera that can generate depth information regarding the distance to the subject.

[0039] In addition, the collection unit (111) can receive sounds from the user (10) or the environment around the user through a sound sensor such as a microphone. Here, since the user's voice or ambient sounds are important information that are treated as user input in interactions with the user, it is preferable that the microphone be placed in a location where there are no elements that interfere with receiving the user's voice or ambient sounds. For example, if the microphone is placed immediately behind the hole of the speaker unit formed on the front of the body unit (110), the user's voice or ambient sounds can be directly received.

[0040] Meanwhile, the microphone may be formed as an integral part of the speaker unit, rather than as an independent component positioned separately from the speaker unit, as described below. In this case, consideration of the microphone placement location described above will not be necessary, and the space utilization within the body unit (110) will also be improved.

[0041] In addition, the collection unit (111) can receive device information generated from another smart device through a communication unit to be described later, and a detailed description thereof will be described later.

[0042] In addition, the collection unit (111) may include one or more sensors for sensing at least one of information within the customized service providing device (100) (in particular, the operating status of the customized service providing device), information on the surrounding environment surrounding the customized service providing device, location information, and user information.

[0043] For example, the collection unit (111) may be equipped with a proximity sensor, a laser scanner (lidar sensor), an RGBD sensor, a geomagnetic sensor, an ultrasonic sensor, an inertial sensor, a UWB sensor, etc. in addition to the above-mentioned sensors.

[0044] The display unit (115) is a type of liquid crystal display device, and can output one or more of text, images, and videos of predetermined information. Here, the predetermined information can include status information of the customized service providing device (100), such as communication status strength information, remaining battery level information, and wireless Internet ON / OFF information. The display unit (115) can display media content that the customized service providing device (100), which will be described later, displays in response to situational information, and can also display information that converts voice output by the customized service providing device (100), which will be described later, through the speaker unit into text. For example, if the customized service providing device (100) sets a wake-up alarm in the process of providing an appropriate service to the user through situational awareness, the text corresponding to this can be output on the display unit (115) as “Wake up at 7:00 AM.”

[0045] The text output on the display unit (115) may be one of the previously described pieces of information repeatedly output, or multiple pieces of information may be output alternately, or specific information may be set as default and output. For example, status information of the customized service providing device (100), such as communication status strength information, remaining battery level information, and wireless Internet ON / OFF information, may be set as small text at the top or bottom of the display unit (115) by default and continuously output, and other information may be output alternately.

[0046] Meanwhile, it is said that the display unit (115) can output one or more of images and videos. In this case, it is preferable to implement the display unit (115) as a liquid crystal display device with a high resolution and a large size, rather than a case where only text is output, in order to improve visibility. The display unit (115) of FIG. 1 should be viewed as an example.

[0047] The speaker unit (116) can output various sounds including voice. Here, voice corresponds to auditory information output by the customized service providing device (100) for interaction with the user, and the type of voice can be set in various ways through direct control of the media-only application installed on the user's terminal (not shown) or the customized service providing device (100).

[0048] For example, you can select the type of voice output through the speaker section from among various voices, such as male voice, female voice, adult voice, and child voice, and you can even select the type of language, such as Korean, English, Japanese, and French.

[0049] Meanwhile, although the speaker unit outputs sound, it does not perform the role of a typical speaker, such as general sound output, and this can also be performed in the same manner. For example, if a user wants to listen to music through the customized service provision device (100), the music can be output through the speaker unit, and if a video is output through the display unit (115), sound synchronized with the video can be output through the speaker unit (116).

[0050] The moving unit (120) can move the customized service providing device (100). More specifically, the moving unit (120) provides a means for the robot's body (110) to move within a specific space in response to a movement command from the control device. More specifically, the moving unit includes a motor and a plurality of wheels, which, when combined, can perform functions of driving, changing direction, and rotating the customized service providing device (100).

[0051] The functions performed by the above-described body part (110) and moving part (120) are performed by control signals output from a control part (not shown) within the body part, but for convenience of explanation, the description will continue as if they were performed by a customized service providing device (100).

[0052] Below, each component of the customized service provision device (100) will be examined.

[0053] FIG. 2 is a configuration diagram of a customized service providing device (100) according to one embodiment of the present invention. Referring to FIG. 2, the customized service providing device (100) may include a body part (110) and a moving part (120), wherein the body part (110) may include a collection part (111), a derivation part (112), a situation recognition part (113), a service providing part (114), a display part (115), a speaker part (116), a communication part (117), and a database (118).

[0054] The collection unit (111) can collect user-related photographic information through a visual sensor. As described above with reference to FIG. 1, the collection unit (111) can include various sensors to collect information about the user's surrounding environment, and can include a camera as a type of visual sensor.

[0055] The body part (110) can be rotated or moved to face the direction in which the user is located through the moving part (120), and through this, the collection part (111) can capture the user through the camera and generate capture information. In addition, the camera of the collection part (111) is also equipped with a control device for controlling the capture direction of the camera, so that the user's entire body can be captured or the capture can be performed so that the user's face is more clearly revealed through the control of the up-down or left-right direction of the camera.

[0056] The captured information may be visual information including an image or video of the user as the subject, and the captured information may also include the user's body and objects or background around the user, which may be utilized in subsequent steps to analyze the user's situation.

[0057] The collection unit (111) can further collect at least one piece of information from among time information on the time at which the photographing information is collected, robot location information generated by recognizing the location of the robot through a location sensor, sound information generated by recognizing the user's voice or sounds generated in the user's surroundings through a sound sensor, and device information generated from another smart device.

[0058] Among these, device information is information that can detect the location of a smart device, connect to the smart device when necessary, and obtain from the smart device, and the customized service providing device (100) can control the moving unit (120) to move to the location where the user is located based on the device information.

[0059] The above smart device may be a user-owned smartphone, smartwatch, or wearable device. Additionally, the smart device may refer to a monitoring device that contains information about the user and is used to monitor the user.

[0060] Smart devices are typically worn, carried, or placed within close proximity to the user, and can generate location information about the smart device via GPS or short-range wireless signals. Therefore, when the customized service provision device (100) receives location information or signals from mutually authenticated or connected smart devices, it can more quickly and easily determine the user's location, thereby controlling the mobile unit (120) to locate the user.

[0061] In addition, device information is information generated by a smart device by detecting the movement of the smart device, and the derivation unit (112) and the situation recognition unit (113) can generate situation information based on the element information and the device information.

[0062] Smart devices can be equipped with internal gyro sensors or proximity sensors to generate movement information based on the user's movements. This movement information can be used to detect sudden falls, changes in posture, or unusual activity. Furthermore, the movement information generated by the smart device can be generated based on the user's movements, such as when the user remains motionless for a long period of time or moves at a constant speed or direction. In other words, the movement information can include information not only on the user's rapid movement, but also on the user's still state, or on the user's movement at a constant speed or direction.

[0063] The situation recognition unit (113) described later can use the above movement information as data to determine whether the user is in a situation requiring an emergency call.

[0064] Additionally, the smart device may be a wearable device attached or worn by the user. In such cases, the smart device can collect various bio-data from the user, such as heart rate, respiration, body temperature, sleep status, blood sugar, blood pressure, stress levels, or dietary status. The bio-data generated from the smart device can be provided to the customized service provision device (100) via wired or wireless means.

[0065] The personalized service providing device (100) can use the received bio-data as data to determine the user's situation. For example, if the user's heart rate or breathing is abnormal compared to the usual state, the personalized service providing device (100) can set the user's situation as urgent when generating situation information to be described later. The derivation unit (112) can input the information collected by the collection unit (111) into at least one analysis model and derive element information for each analysis model. As described above, the collection unit (111) can collect photographing information of the user, sound information of the user or the surrounding environment, location information of the user, and device information through at least one equipped sensor, and the information collected in this way can be provided to the analysis model of the derivation unit (112).

[0066] The derivation unit (112) may include at least one analysis model, and each analysis model may perform analysis on predetermined input data to derive predetermined output data for each analysis model. The output data output by each analysis model may then be used as individual element information constituting integrated information.

[0067] More specifically, the analysis model may include at least one of a spatial classification model for determining the type of space within an image included in the shooting information, an object detection model for determining the location and type of an object within an image included in the shooting information, and an action recognition model for determining the posture of a user within an image included in the shooting information.

[0068] Additionally, the analysis model may include a face detection model for detecting a user's face within an image included in the shooting information and determining the location of the face.

[0069] In addition, the analysis model may include at least one of an identity recognition model for recognizing a user's identity from a facial image generated based on a position of the determined face, an expression recognition model for recognizing a user's facial expression from a facial image generated based on a position of the determined face, and a face direction recognition model for recognizing a face direction from a facial image generated based on a position of the determined face.

[0070] Hereinafter, each analysis model will be described in more detail with reference to FIGS. 3 to 10, 11a, and 11b.

[0071] Figure 3 is a diagram illustrating a spatial classification model among the analysis models included in the derivation unit (112). The spatial classification model may be a classification model based on a known convolutional neural network (CNN). For this purpose, CNN architectures such as AlexNet, VGG-16, Inception, ResNet, and MobileNet may be used.

[0072] As shown in Figure 3, the spatial classification model extracts a feature map containing features within an image by repeating convolution and subsampling operations. This feature map is then passed through a fully connected layer to output a score vector for each space. The space with the highest score among the preset values ​​for multiple spaces can be considered the space ultimately recognized by the spatial classification model.

[0073] For example, living room, kitchen, bedroom, and veranda can be exemplified as space classes that can be output through a space classification model, and when the derivation unit (112) applies the input shooting information to the space classification model and the score for the living room is derived as the highest, the derivation unit (112) can generate a result value indicating that the space where the user is currently located is the living room as element information.

[0074] In order to improve the accuracy of spatial classification using a spatial classification model, when the robot draws a map of the space where the user is located, spatial information and captured images can be used together for training the spatial classification model.

[0075] FIG. 4 is a diagram illustrating an object detection model among the analysis models included in the derivation unit (112). The object detection model may utilize a classification model based on a convolutional neural network, and may output a bounding box indicating the type of the detected object and the area in which the object is located. For this purpose, a convolutional neural network architecture such as AlexNet, VGG-16, Inception, ResNet, and MobileNet may be used. Representative object detection models may include RCNN, Fast RCNN, YOLO, Single Shot Detector (SSD), Retina-Net, and Pyramid Net.

[0076] As shown in Fig. 4, the object detection model can generate multiple bounding boxes for the areas where objects are located in the image through convolution operations. Thereafter, the object detection model can derive the final bounding box by applying the NMS (Non-Maximum Suppression) operation, which removes all bounding boxes except the bounding box with the highest score for the multiple bounding boxes and the image, and can analyze the characteristics of the objects in the bounding boxes to determine which objects they correspond to. In the case of Fig. 4, when the image input to the object detection model includes multiple vehicles driving on the road, the model generates bounding boxes for the areas where the vehicles are located, recognizes that the objects are vehicles, and outputs the type and reliability of the object, and the location and size of the bounding boxes as the result values ​​of the object detection model as the final output.

[0077] To improve accuracy, object detection models can be trained to output only objects likely to be within the robot's range of motion. For example, for a robot with a range of motion set within a home, the object detection model can be trained to recognize only objects such as people, vacuum cleaners, TVs, and beds.

[0078] Figures 5 and 6 are diagrams illustrating an action recognition model for determining a user's pose in an image included in shooting information. Since the process of recognizing the user's pose must come first in order to recognize the user's action, the pose estimation model is first explained through Figure 5. For skeletal extraction, a convolutional neural network-based skeletal extraction algorithm can be used on a 2D or 3D image to extract 2D / 3D skeletal information. A convolutional neural network-based feature point detection model can be applied as a skeletal information extraction model, and for example, algorithms such as MoveNet, PoseNet, OpenPose, MediaPipe, AlphaPose, Nuitrack, and Kinecct SDK can be utilized. Skeletal information and values ​​calculated based on it do not change in a short period of time, so they can be effectively used to identify a tracking target.

[0079] Referring to Figure 5, the skeletal extraction model can receive 2D or 3D images and extract 2D or 3D coordinates and their reliability for each joint. The extracted joint data may be in the form of a graph, with nodes representing the human body, such as the head, shoulders, chest, arms, hips, and legs, and each node connected by edges.

[0080] To improve the accuracy of results, the skeletal extraction model can extract joint information by selecting a model that extracts only the skeleton of a single person or a model that extracts the skeletons of multiple people, depending on the type of service provided by the robot. For example, for a one-to-one service, the robot can use a model that extracts only the skeleton of a single person, while for a service serving multiple people in a household, a model that can extract multiple skeletons can be used. Furthermore, the skeletal extraction model can be selected as a single-person or multiple-person version depending on the number of people detected by the object detection model.

[0081] Figure 6 is a diagram illustrating an example of an action recognition model for analyzing what kind of action is taken from joint data generated through a skeletal extraction model.

[0082] When inputting an image or video into an artificial intelligence model without preprocessing, there is a problem that the processing speed is slow due to the large size of the data to be processed. Therefore, as mentioned above, it is desirable to apply a skeleton extraction model to the input image to extract joint data, and then recognize what action is performed based on this.

[0083] Action recognition models can recognize actions using 2D / 3D skeletal information extracted from 2D / 3D images using a skeletal extraction model. For example, a rule-based method that classifies actions using a threshold-based method can be used. Alternatively, a machine learning-based method can be used, such as a recurrent neural network (RNN), which is a powerful neural network for processing time-series data and suitable for action recognition based on time-series skeletal information, or a graph convolutional neural network (GCN) that utilizes the graph structure of skeletal data.

[0084] Referring to Fig. 6, it can be seen that the process of extracting joint data by applying a skeletal extraction model to an input image and then recognizing the actions of a person in the image by applying an action recognition model based on skeletal information can be seen.

[0085] To improve the accuracy of results, action extraction models can limit the types of actions they can recognize. For example, they can limit the types of actions they output to only actions primarily performed inside the home.

[0086] Figure 7 is a diagram illustrating an example of a clothing recognition model for analyzing what kind of clothing is worn from joint data generated through a skeletal extraction model.

[0087] Joint data obtained through the skeletal extraction model can be used to identify the positions of a person's shoulders and ankles within the image, allowing for cropping of the top and bottom clothing worn by the person. The clothing recognition model uses a convolutional neural network-based model to digitize clothing information and extract its features.

[0088] Referring to Figure 7, it can be seen that a skeletal extraction model is applied to a photographed image of a person to extract joint data, and this joint data is input into a clothing recognition model to extract feature values ​​and store them in a database. Thereafter, when a new image is input, the feature values ​​extracted by applying the skeletal extraction model and clothing recognition model are compared and matched with the feature values ​​stored in the existing database, thereby recognizing an image containing a person wearing the same clothing.

[0089] FIG. 8 is a drawing for explaining an example of a face detection model for detecting a user's face in an image included in shooting information and determining the location of the face.

[0090] The face detection model can be used in a similar way to the object detection model described above. Unlike object detection models that detect specific objects within an image, the face detection model can identify the location of faces within an image and generate bounding boxes corresponding to those locations. The face detection model can be applied to a convolutional neural network (CNN)-based detection model, and object detection algorithms such as RCNN, Fast RCNN, YOLO, Single Shot Detector (SSD), Retina-Net, and Pyramid Net can be utilized.

[0091] Referring to Figure 7, it can be seen that the face detection model extracts the location of at least one human face within the image and generates a bounding box as output. The bounding box generated by the face detection model can be used as input data for the analysis models described below, such as the identity recognition model, facial expression recognition model, and facial orientation recognition model.

[0092] To improve the accuracy of the results, the face detection model can be fed with information about the user's height, either pre-entered or estimated using a skeletal model. This allows the robot to rotate the camera unit vertically to face the user's face, enabling accurate recognition of the user's face even at close range while standing.

[0093] FIG. 9 is a diagram illustrating an identity recognition model for recognizing a user's identity from a facial image generated based on the position of the identified face.

[0094] Feature points can be extracted from the bounding box image extracted by the identity recognition model and stored in a database. By comparing the feature points extracted from the bounding box image with the feature points already stored in the database, the face of the target can be identified. The identity recognition model can be a convolutional neural network-based model, and examples of algorithms include VGG-Face, FaceNet, OpenFace, DeepFace, and ArcFace.

[0095] Referring to Figure 9, it can be seen that a face detection model is applied to an image of a person, a bounding box is created for the area where the face is located, and a face image is extracted. This face image is then input into an identity recognition model, where its feature values ​​are extracted and stored in a database. Thereafter, when a new image is input, the aforementioned face detection model and identity recognition model are applied, and the extracted feature values ​​are compared and matched with the feature values ​​stored in the existing database to recognize the identity.

[0096] To improve the accuracy of the results, the identity recognition model can provide voice guidance to the user, directing their face toward the robot's camera. This allows the robot to generate an image of the user's face from the front, enabling more accurate identification. Whether the user's face is facing the camera can be confirmed through the resulting data generated by the face orientation recognition model described below.

[0097] FIG. 10 is a diagram illustrating an expression recognition model for recognizing a user's facial expression from a facial image generated based on the position of the determined face.

[0098] The facial expression recognition model can extract only the facial image through a face detection model and then input the extracted facial image. The facial expression recognition model uses a classification model based on a convolutional neural network and outputs scores for one or more specified facial expressions. The facial expression recognition model can determine the expression with the highest score among multiple expressions as the final recognized expression. The facial expression recognition model can utilize one or more convolutional neural network architectures, such as AlexNet, VGG-16, Inception, ResNet, and MobileNet.

[0099] To improve accuracy, facial expression recognition models can be trained using facial expression data collected from a specific dataset. Furthermore, user facial expression data captured by the robot can be used during the training process. To achieve this, the robot can provide voice guidance to the user, encouraging them to make various facial expressions.

[0100] Figures 11a and 11b are drawings illustrating a face direction recognition model for recognizing a face direction from a face image generated based on the position of the determined face.

[0101] The face orientation recognition model can extract landmarks from a face image extracted through a face detection model. This process can utilize a convolutional neural network, and the face orientation recognition model can output results indicating the direction the face is facing based on eye and nose positions and facial contours.

[0102] Referring to Figure 11a, it can be seen that a bounding box for a face image is generated from an input image through a face detection model, and the face image is included in a face direction recognition model to extract facial landmarks. Referring to Figure 11b, it can be seen that the process of outputting a result value indicating the direction in which the face is facing from the extracted facial landmarks can be seen.

[0103] Returning to FIG. 2, the derivation unit (112) can generate element information through the above-described analysis model. Here, element information may refer to information including the result values ​​output by each analysis model in response to input data.

[0104] The element information generated from each analysis model can form a single integrated information. That is, the integrated information can comprehensively store the values ​​analyzed through the aforementioned analysis model. For example, the integrated information, which combines each element information, can yield information such as "the user is in the living room," "the user is moving his or her arm," "the user has a neutral expression," and "the user is staring straight ahead."

[0105] More specifically, element information and integrated information can be defined as follows. Among element information, user location information can refer to the user's location on an indoor map drawn by a robot. Here, location information can refer to two-dimensional coordinate values ​​based on a specific point on the indoor map.

[0106] Among element information, spatial information can be derived from a spatial classification model, as it classifies the space in which the user is currently located. Furthermore, spatial information can also include a probability value indicating the user's location in that space. For example, spatial information might include information such as a 90% probability that the user is located in the living room in a captured image.

[0107] Among the element information, object information can be derived from an object detection model to classify an object detected in the robot's field of view. Furthermore, object information can include a probability value indicating that an object in a captured image belongs to a given category, and can also include a result value indicating whether the object is being held by the user. For example, the object information can include information such as an object in a captured image being classified as a TV, an 80% probability that it is a TV, and the user is not holding a TV. As another example, the object information can include information such as an object in an image being classified as a vacuum cleaner, a 99% probability that it is a vacuum cleaner, and the user is holding a vacuum cleaner.

[0108] Among the element information, behavioral information can be derived from an action recognition model to classify the behavior of a user detected in the robot's field of view. Furthermore, the behavioral information can include a probability value that the user's behavior in the captured image falls into a given category. For example, the behavioral information can include information such as the user in the captured image moving his or her hand up and down, with an 80% probability of such behavior. Another example is the user in the captured image bending down, with a 70% probability of such behavior.

[0109] Among the element information, time information is information about the current time, and this can be generated by the above-mentioned collection unit (111).

[0110] Among the element information, identity information is information for classifying the identity of a user detected in the robot's field of view, and can be derived by an identity recognition model. Furthermore, the identity information may include a probability value that a user in a captured image corresponds to a classified identity, and personal information loaded from a database (118) based on the classified identity. For example, the identity information may include information such as the user in the captured image being named Hong Gil-dong, a 90% probability that the user is Hong Gil-dong, and personal information about Hong Gil-dong, such as his gender being male and his medical condition being Parkinson's disease.

[0111] Among the element information, facial expression information is used to classify the user's facial expression detected in the robot's field of view and can be derived using a facial expression recognition model. Furthermore, facial expression information may include a probability value corresponding to the classified expression of the user in the captured image. For example, the facial expression information may include information such as the user's facial expression in the captured image being happy, indicating a 90% probability of the expression being happy.

[0112] FIG. 12 is a diagram illustrating a process in which a situation recognition unit (113) of a customized service provision device (100) recognizes a user's situation. Referring to FIG. 12 , it can be seen that the situation recognition unit (113) recognizes the current situation through various information obtained from element information (e.g., there is an "object" corresponding to "furniture" around the user; there is a "tool" that the user is "wearing on the body"; the user is "located" in the "living room") and generates corresponding situation information.

[0113] The situation recognition unit (113) can recognize the user's current situation and generate situation information based on integrated information that integrates the derived element information. More specifically, the situation recognition unit (113) can generate situation information from the integrated information using at least one of a machine learning model and a rule-based model.

[0114] Here, contextual information can mean information generated based on element information about the current situation the user is in or the action being performed.

[0115] For example, contextual information such as "the user is eating in the kitchen" can be generated based on integrated information that includes element information such as "the user is in the kitchen," "the user's arm is moving," and "food is detected around the user." Furthermore, contextual information such as "the user is exercising in the living room" can be generated based on integrated information that includes element information such as "the user is in the living room" and "the user's entire body is moving repeatedly."

[0116] The situation information generated by the above method can be matched with the integrated information input into the situation recognition unit (113) and stored in the database (118).

[0117] A rule-based model may be a model in which a designer inputs situation information corresponding to element information in advance as a rule, and when element information matching the pre-input rule is input, situation information is generated according to the rule.

[0118] For example, if the element information can be used to determine that the user is holding a vacuum cleaner and moving an arm, the situation recognition unit (113) can generate situation information indicating that the user is cleaning according to preset rules.

[0119] On the other hand, a machine learning model can be a model created by preprocessing the element information that constitutes integrated information, then inputting the data into an artificial neural network and training it to output contextual information.

[0120] When the situation recognition unit (113) generates situational information based on a machine learning model, the user's location, actions, identity, and facial expressions, as identified through element information, can be stored in a database and utilized as learning data. The element information input to the machine learning model can be input to the machine learning model for learning through a data preprocessing process that vectorizes the output values ​​according to each analysis model, and can be trained to generate more accurate situational information through supervised learning.

[0121] Additionally, the situation recognition unit (113) can generate routine information based on integrated information and situational information related to the user's daily routine, if the user's daily routine exhibits a pattern that repeats according to a certain standard through data stored in the database (118), and can utilize the routine information to generate more accurate situational information. For example, if the user's tendency to eat after showering is observed repeatedly every day or every week, this pattern can be learned to generate situational information corresponding to the user's meals.

[0122] The process of generating situation information by the situation recognition unit (113) and the process of generating situation information by additionally utilizing daily routine information will be explained with reference to the following Figures 13 and 14.

[0123] FIG. 13 is a drawing for explaining that a customized service providing device according to one embodiment of the present invention provides a customized service.

[0124] Referring to Fig. 13, it can be seen that a robot (1301), such as a customized service providing device, photographs a user and generates photographing information (1302). Thereafter, the photographing information (1302) can be input into a spatial classification model (1303), an object detection model (1304), a skeleton extraction model (1305), an action recognition model (1306), a face detection model (1307), an identity recognition model (1308), and an expression recognition model (1309), thereby generating element information including result values ​​for each analysis model.

[0125] Next, it can be seen that integrated information (1312) is generated by integrating element information, user location information (1310), and time information (1311). Thereafter, it can be seen that the integrated information (1312) is input into the situation recognition unit (1313) to generate situation information (1314) about the current situation, and that service provision (1315) is performed based on the situation information (1314).

[0126] Figure 14 is a diagram illustrating a customized service providing device according to one embodiment of the present invention providing a customized service. Unlike Figure 13, Figure 14 demonstrates that additional pattern information is generated using information stored in a database.

[0127] Referring to Fig. 14, it can be seen that a robot (1401), such as a customized service provision device, photographs a user and generates photographing information (1402). Thereafter, the photographing information (1402) can be input into a spatial classification model (1403), an object detection model (1404), a skeleton extraction model (1405), an action recognition model (1406), a face detection model (1407), an identity recognition model (1408), and an expression recognition model (1409), thereby generating element information including result values ​​for each analysis model.

[0128] Next, it can be seen that integrated information (1412) is generated by integrating element information, user location information (1410), and time information (1411). Thereafter, it can be seen that the integrated information (1412) is input into the situation recognition unit (1413) to generate situation information (1414) about the current situation, and that the situation information (1414) and the integrated information (1412) are stored in a database (1415). Through the information stored in the database (1415), usual routine information (1416) that patterns the user's usual routine can be generated, and the usual routine information (1416) can be input into the situation recognition unit (1413) together with the integrated information (1412) to be used to generate more accurate situation information (1414). Thereafter, it can be seen that a service is provided (1417) based on the situation information (1414).

[0129] The process of generating situation information using at least one of a rule-based model and a machine learning model by the situation recognition unit (113) is described in more detail with reference to FIGS. 15 to 18.

[0130] FIG. 15 and FIG. 16 are drawings for explaining by example that a situation recognition unit (113) according to an embodiment of the present invention generates situation information according to a rule-based model.

[0131] A rule-based model can determine detailed elements of a situation (time, location, identity, behavior, emotion, importance) based on pre-set conditions through element information. The situation recognition unit (113) can generate situation information using element information with a probability value greater than a threshold value based on threshold criteria set differently for each element information.

[0132] Contextual information may include information about the current time and location of the user, the user's identity, the user's actions, the user's current emotions, and the importance of the current situation.

[0133] The context recognition unit (113) can use multiple element information to construct context information. For example, information about the user's actions among the user's context information can be derived based on the element information including location information, object information, action information, and time information. Furthermore, the importance of the user's context information can be derived based on the element information including spatial information, action information, time information, identity information, and facial expression information.

[0134] The above rule-based model may correspond to a model in which a designer sets rules for element information used to configure the above-described situation information and threshold criteria for each element information.

[0135] Referring to FIG. 15, it can be seen that location information, spatial information, object information, action information, time information, identity information, and expression information are input into the situation recognition unit (113) as element information (1501) constituting the integrated information. The situation recognition unit (113) determines whether the probability value included in the above element information satisfies the threshold criterion, and then can derive from the element information satisfying the threshold criterion that the current user is located at the coordinates x, y = 3.4, 2.1, the type of place where the user is located is a living room, a vacuum cleaner is also recognized, the user's arm movement is recognized, the current time is 2:00 PM, and the user's identity information is that the name is Hong Gil-dong, he is a 50-year-old male, he has no illness, and the user's expression is expressionless. The situation recognition unit (113) can generate situation information (1502) such as "Hong Gil-dong is cleaning the living room at 2:00 PM. He has no special emotions and is a low-importance situation" through the derived information. Here, you can see what element information is involved in the creation of each piece of situational information (time, place, identity, action, emotion, importance) through the connection of arrows.

[0136] When generating context information through a rule-based model, if a specific element information does not satisfy the threshold criterion, the element information cannot be used to generate context information due to uncertainty. If there is a large amount of element information that does not satisfy the threshold criterion, generating context information may also be difficult. The context recognition unit (113) can enable the robot to provide the service if it is possible to provide the service even if there is element information that does not satisfy the threshold criterion. On the other hand, if the context recognition unit (113) cannot provide the service due to element information that does not satisfy the threshold criterion, the robot can obtain the relevant information from the user through voice conversation or other communication methods. Alternatively, the threshold criterion set for the relevant element information can be relaxed to use the next best result as information. Alternatively, context information can be generated using information from daily routines.

[0137] Referring to Fig. 16, unlike the case of Fig. 15, an example is provided where the probability value for facial expression information among the element information (1601) does not satisfy the threshold criterion. In this case, the situation recognition unit (113) cannot recognize the user's facial expression, and thus cannot generate information about the user's emotions among the situation information.

[0138] Rule-based models can improve the accuracy of action recognition by setting different thresholds for each element of contextual information. Among the detailed elements that comprise contextual information (time, location, identity, action, emotion, and importance), the most important is information about the action itself. Therefore, by setting different elements of importance for each action, action recognition accuracy can be improved. In other words, because the available element information for each contextual action varies, the element information used can be set differently for each action.

[0139] For example, if a user is holding a vacuum cleaner, context information can be generated as if the user is cleaning. In other words, when recognizing a cleaning action, the importance of object information can be increased by changing the threshold criteria. If the user is holding a vacuum cleaner in the living room at 2 PM, or if the user is holding a vacuum cleaner at any time, or if the user is walking around holding a vacuum cleaner, context information indicating that the user is currently cleaning can be generated based on the element information indicating that the user is holding a vacuum cleaner. In this case, even if other element information does not meet the threshold criteria, if the object information among the element information indicates that the user is holding a vacuum cleaner, context information can be generated for the user.

[0140] As another example, the context recognition unit (113) can generate context information indicating that a user is sleeping if the user is in bed between midnight and morning. For example, if the user is lying in bed at 2:00 AM, context information indicating that the user is sleeping can be generated based on time and location information even if the behavioral or emotional information among the element information does not meet the threshold criteria.

[0141] As another example, the situation recognition unit (113) can generate situation information that the user is washing dishes if the user is located in the kitchen and the sound of water is heard around the user, even if other element information does not meet the threshold criteria.

[0142] As described above, the situation recognition unit (113) can also generate situation information by using the user's daily routine information and integrated information. Here, the daily routine information may be an arrangement of likely situations that may occur in the present in order of likelihood (e.g., exercise as the first priority, nap as the second priority).

[0143] The situation recognition unit (113) can also generate routine information to list situations in order of frequency of occurrence at a specific time during a specific period (e.g., 7 times of exercise, 5 times of nap, and 4 times of walk between 2 PM and 3 PM during a week).

[0144] The situation recognition unit (113) can also generate routine information to list situations in order of frequency that occurred at a specific location at a specific time during a specific period (e.g., when in the living room at 2 p.m. during a week, exercise occurred 7 times, TV was watched 4 times, and cognitive games were played 2 times).

[0145] Additionally, the situation recognition unit (113) can generate routine information by calculating the frequency with which situations occur after a specific situation during a specific period (e.g., after exercising, showers occurred 4 times, meals occurred 2 times, and walks occurred 1 time in a week).

[0146] The situational awareness unit (113) can improve the accuracy of situational awareness through situational information by generating situational information by adding the above-mentioned usual work information to the integrated information.

[0147] FIG. 17 and FIG. 18 are drawings for explaining by example that a situation recognition unit (113) according to an embodiment of the present invention generates situation information according to a machine learning model.

[0148] A machine learning model can be created by learning the various rules required for a rule-based model and the contextual information generated accordingly. Training data for training a machine learning model can be manually recorded by the designer or collected through a rule-based model. The input data for training a machine learning model can include classification results, including spatial information, object information, behavioral information, identity information, and facial expression information, as well as probability values.

[0149] When using machine learning-based models, weights can be learned that reflect the importance or influence each element has on contextual information. This eliminates the need to set thresholds for each element, as is the case with rule-based models. In other words, machine learning-based models can automatically determine important element information for each action through learning.

[0150] Referring to FIG. 17, it can be seen that each element information (1701) is input into a machine learning model (1702), and situation information (1703) is output from the machine learning model (1702). Referring to FIG. 18, it can be seen that each element information (1801) is input into a machine learning model (1802), but information on time, location, and identity among the situation information is directly extracted and generated from the time information, location information, space information, and identity information among the element information without using the output of the machine learning model.

[0151] FIG. 19 is a drawing for explaining by example that a situation recognition unit (113) according to an embodiment of the present invention generates situation information by utilizing a rule-based model and a machine learning model together.

[0152] The situation recognition unit (113) can generate situation information using one of the rule-based model and machine learning model described above, but can also generate situation information using both models.

[0153] The first embodiment in which the situation recognition unit (113) generates situation information using both a rule-based model and a machine learning model derives the results of both the rule-based model and the machine learning model, but sets a condition for which model's result is to be used for detailed element information whose results are different from each other.

[0154] More specifically, in terms of detailed element information constituting situational information, information on time, place, identity, and emotion may yield the same results for rule-based models and machine learning models. This is because, regardless of the model used when the situation recognition unit (113) infers time, place, identity, and emotion among the detailed element information, it references the same element information (e.g., time information, place information, identity information, and facial expression information among the element information), so there is no difference in the resulting values ​​when using the two models.

[0155] However, because detailed information about different situations (e.g., information about actions among situational information) may produce different results when using rule-based models and machine learning models, you can set conditions for which model's results are used.

[0156] As an example of such conditions, threshold criteria can be utilized. For example, in the case of a machine learning model, classification results and probability values ​​can be derived from information on actions and importance among the detailed information of situational information. Therefore, a threshold criterion for information on actions among the detailed information can be set so that the classification results and probability values ​​are derived only when the probability value is higher than the threshold criterion, and the results of the rule-based model are used when the probability value of the machine learning model is lower than the threshold criterion.

[0157] For example, suppose a rule-based model generates contextual information, producing results like "Mr. Hong Gil-dong is cleaning the living room at 2 PM (no emotion, medium importance)", while a machine learning model generates results like "Mr. Hong Gil-dong is walking around the living room at 2 PM (no emotion, low importance)". In this case, the two models yield identical results for information about time, location, and identity, among the detailed information that constitutes contextual information, while information about actions and importance yield different results.

[0158] In determining information about an action among the detailed element information in the example described above, the situation recognition unit (113) derives a probability value of the information about the action (“walking”) among the situation information generated through the machine learning model, and if the probability value is lower than the threshold standard, the information about the action (“cleaning”) among the situation information generated through the rule-based model can be used.

[0159] Similarly, when the situation recognition unit (113) determines the information on importance among the detailed element information in the above example, the probability value of the information on importance (“low importance”) among the situation information generated through the machine learning model may be derived, and if the probability value is higher than the threshold standard, the result of the machine learning model may be used instead of the result of the rule-based model (“medium importance”). In this way, the situation information ultimately generated by the situation recognition unit (113) may be as follows: “At 2 PM, Mr. Hong Gil-dong cleans the living room (no emotion, low importance).”

[0160] The second embodiment in which the situation recognition unit (113) generates situation information using both a rule-based model and a machine learning model is to set in advance, for each piece of detailed element information that constitutes the situation information, detailed element information that follows the result of the rule-based model and detailed element information that follows the result of the machine learning model.

[0161] Referring to FIG. 19, it can be seen that among the detailed element information constituting the situation information (1903), information on time, place, identity, and action is set to follow the result of the rule-based model (1901), and information on emotion and importance is set to follow the result of the machine learning model (1902).

[0162] These setting criteria are exemplary, and each detailed element information can be set to follow the results of other models, depending on the designer's intention.

[0163] FIG. 20 is a drawing for explaining that a customized service providing device according to one embodiment of the present invention provides a customized service.

[0164] Referring to FIG. 20, unlike the case of FIG. 13 described above, it can be seen that recording information (2016) is generated from the robot (2001), this recording information (2016) is input into the sound classification model (2017), and the output of the sound classification model (2017) is included in the integrated information (2012).

[0165] As described above, the robot (2001) can record sounds from the user or the robot's surroundings through a microphone included in the body. The recorded sounds can then be converted into digital data through a signal conversion process of the microphone. The recording information (2016) can have time-series waveform data and can be input into a sound classification model (2017) and analyzed by an artificial intelligence algorithm for classifying sound data.

[0166] The sound classification model (2017) can measure the size of a specific frequency band in sound or extract sound energy or pitch using frequency analysis through a feature extraction algorithm utilizing machine learning. In addition, the sound classification model (2017) can apply a machine learning algorithm that classifies sounds based on the features extracted through the classification algorithm. Such an algorithm can generally be implemented using a supervised learning method. That is, it learns pre-classified sound data and classifies new sound data. Through the sound classification model (2017), it is possible to classify whether the sound currently input through the microphone is the user's voice, the sound of washing dishes, the user's breathing, or the sound of a vacuum cleaner. The type of sound classified in this way and its probability value can be generated as sound information, which is one of the element information. The sound information can then be integrated with other element information and utilized as integrated information (2012).

[0167] Thereafter, the context recognition unit (2013) can generate context information based on other element information and sound information. For example, if the context recognition unit determines that the user is in the kitchen and the sound information indicates the sound of water, it can determine that the user is washing dishes based on the action information within the context information. As another example, if the user is walking around and the sound information indicates the sound of a vacuum cleaner, it can determine that the user is cleaning based on the action information within the context information.

[0168] The situation recognition unit (113) can generate situation information by considering not only element information but also the above-mentioned device information.

[0169] For example, when device information is received that a smartphone being carried by a user is falling rapidly, the situation recognition unit (113) can generate situation information indicating an emergency situation based on the element information that the user's location is a bedroom and the element information that the user's facial expression is a pained expression.

[0170] For another example, when a wake-up notification signal is received from a smartphone carried by a user or movement of a smartphone of a user who is sleeping is detected, the situation recognition unit (113) can generate situation information indicating that the user has finished sleeping and has woken up.

[0171] The service provider (114) can provide customized services to the user based on situational information. More specifically, the service provider (114) can provide at least one of a service that outputs images based on situational information, a service that outputs audio based on the situational information, and a service that makes an emergency call to a preset device based on the situational information.

[0172] The service provider (114) can provide appropriate services to users by utilizing various output devices provided in the body (110) of the customized service provider (100). For example, the service provider (114) can play media content through the display unit (115) and output voice or voice guidance related to the content being played through the speaker unit (116). In addition, the service provider (114) can transmit wireless signals to other terminal devices, a router hub, or a server through the communication unit (117).

[0173] Examples of services provided by the service provider (114) based on situation information are as follows.

[0174] The service provider (114) can play songs or videos while the user is eating in the kitchen. The service provider (114) can play songs or videos while the user is exercising in the living room. The service provider (114) can play songs or videos while the user is cooking or washing dishes in the kitchen.

[0175] The service provider (114) can output voice guidance recommending exercise or walking or display related text when the user sits on the sofa for a long time.

[0176] If a user does not move in one place for a long time, the service provider (114) can provide voice guidance encouraging the user to move or display related text.

[0177] The service provider (114) can execute emotional management content when a user sits on a sofa and shows a depressed expression. The service provider (114) can visit the user at the time of taking medication and display a notification that it is time to take the medication. Afterwards, the service provider (114) can observe the user's behavior to determine whether the user has completed taking the medication, and control the system to store the relevant information in a database or notify the server via the communication unit (117).

[0178] The service provider (114) can output a fall warning if the user is lying in a fall-risk area. Furthermore, the service provider (114) can check the facial expressions and behavior of the fallen user, and if it determines that the user is unconscious, it can transmit a signal via the communication unit (117) to the guardian's terminal or a preset emergency rescue organization.

[0179] The service provider (114) can check the user's facial expressions or actions when the user is lying down in a place where the user does not usually lie down, and if it is determined that the user is unconscious, it can transmit a signal to the guardian's terminal or a preset emergency rescue organization through the communication unit (117).

[0180] The service provider (114) can notify the user of the day's schedule when the robot wakes up in the morning. If the robot is determined to be in an inaccessible location (e.g., a closed bathroom) for a specified period of time or in an unusual time pattern, the service provider can provide voice guidance to the user to confirm the current situation. After this, an emergency call can be made based on the user's response.

[0181] The service provider (114) can check the user's situation by outputting voice guidance when the user performs repetitive actions in a specific location, and can perform an emergency call when it is determined to be an emergency situation or when it detects the user leaving the living space according to the user's settings.

[0182] The service provider (114) can use vision recognition information to provide confirmation and warning to the user and, if necessary, make an emergency call when the movement (gait pattern, etc.) indoors is significantly different from usual.

[0183] In addition, the service provider (114) can perform an interaction process with the user through situational awareness information. For example, if the customized service providing device (100) determines the user's emotion through the generated situational information, the customized service providing device (100) can induce the user to interact with the customized service providing device (100) by outputting a motion, voice, or video corresponding to the user's emotion. In addition, if it is detected that the user reacts with voice, facial expression, or action in response to the motion, voice, screen, or video output of the customized service providing device (100), new situational information is generated, enabling the provision of additional customized services.

[0184] Figure 21 is a flowchart illustrating a method for providing customized services according to one embodiment of the present invention.

[0185] Referring to FIG. 21, a method for providing a customized service according to an embodiment of the present invention may include a movement step (S100) of controlling a mobile device to move to a location where the user can be recognized, a collection step (S200) of collecting photographing information related to the user through a visual sensor, an analysis step (S300) of inputting the photographing information into at least one analysis model to derive element information for each analysis model, a situation recognition step (S400) of integrating the derived element information for each analysis model to generate integrated information, recognizing the current situation of the user based on the integrated information to generate situation information, and a service provision step (S500) of providing a customized service to the user based on the situation information.

[0186] Here, the collection step (S100) may be a step of further collecting at least one piece of information from among time information about the time at which the photographing information is collected, robot location information generated by recognizing the location of the place where the robot is located through a location sensor, sound information generated by recognizing the user's voice or sounds generated in the user's surroundings through a sound sensor, and device information generated from another smart device.

[0187] In addition, the device information is information generated by the smart device by detecting the location of the smart device, and the robot can control the moving part to move to the location where the user is located based on the device information.

[0188] On the other hand, the device information is information generated by the smart device by detecting movement of the smart device, and the situation recognition step (S400) may be a step of generating the situation information based on the element information and the device information.

[0189] The analysis model may include at least one of a space classification model for determining the type of space within an image included in the above shooting information, an object detection model for determining the location and type of an object within an image included in the above shooting information, and an action recognition model for determining the posture of the user within an image included in the above shooting information.

[0190] The situation recognition step (S400) may be a step of recognizing the current situation of the user through a machine learning model or a rule-based model to generate the situation information. If element information corresponding to the rule-based model exists, the rule-based model is applied to generate the situation information. If element information corresponding to the rule-based model does not exist, the machine learning model is applied to generate the situation information.

[0191] The service provision step (S500) may be a step of providing at least one of a service that outputs a video based on the situation information, a service that outputs a voice based on the situation information, and a service that makes an emergency call to a preset device based on the situation information.

[0192] The above-described method for providing customized services can be further divided into additional steps or combined into fewer steps, as described in the embodiments described above with reference to FIGS. 1 through 13. Furthermore, some steps may be omitted as needed, or the order of steps may be reversed.

[0193] The above-described method for providing customized services can also be implemented in the form of a computer program stored on a computer-readable recording medium executed by a computer, or a recording medium containing computer-executable commands. Furthermore, the above-described method for scoring sleep quality can also be implemented in the form of a computer program stored on a computer-readable recording medium executed by a computer.

[0194] A computer-readable recording medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, a computer-readable recording medium may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0195] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0196] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims

1. In a device that provides customized services to users, A moving part controlled to move to a place where the user can be recognized; A collection unit that collects shooting information related to the user through a visual sensor; A derivation unit that inputs information collected through the above collection unit into at least one analysis model and derives element information for each analysis model; A situation recognition unit that generates integrated information by integrating element information of the derived analysis model and recognizes the current situation of the user based on the integrated information to generate situation information; and A customized service providing device characterized by including a service providing unit that provides a customized service to the user based on the above situation information.

2. In paragraph 1, The above collection section A customized service providing device characterized in that it further collects at least one piece of information from among time information on the time at which the above-mentioned shooting information is collected, robot location information generated by recognizing the location of the place where the robot is located through a location sensor, recording information generated by recognizing the user's voice or sounds occurring in the user's surroundings through a sound sensor, and device information generated from another smart device.

3. In paragraph 2, The above device information is Information generated by the smart device by detecting the location of the smart device, A customized service providing device characterized in that the customized service providing device controls the moving unit to move to the location where the user is located based on the device information.

4. In paragraph 2, The above device information is Information generated by the smart device by detecting the movement of the smart device, A customized service providing device characterized in that the above situation recognition unit generates the situation information based on the element information and the device information.

5. In paragraph 1, The above analysis model is A customized service providing device characterized by including at least one of a space classification model for determining the type of space in an image included in the above shooting information, an object detection model for determining the location and type of an object in an image included in the above shooting information, and a behavior recognition model for determining the posture of the user in an image included in the above shooting information.

6. In paragraph 1, The above analysis model is Including a face detection model for detecting the user's face in the image included in the above shooting information and determining the location of the face, A customized service providing device characterized in that it further includes at least one of an identity recognition model for recognizing the identity of the user from a facial image generated based on the determined location of the face, an expression recognition model for recognizing the facial expression of the user from a facial image generated based on the determined location of the face, and a face direction recognition model for recognizing the direction of the face from a facial image generated based on the determined location of the face.

7. In paragraph 1, Further comprising a pattern information generation unit that generates pattern information on the user's repetitive work from a database that stores the integrated information and the situation information generated in response to the integrated information; A customized service providing device characterized in that the above situation recognition unit recognizes the current situation of the user based on the integrated information and the pattern information and generates situation information.

8. In paragraph 1, The above situational awareness unit A rule-based model that determines situation information based on whether element information from each of the above analysis models satisfies a predetermined condition. A customized service providing device characterized in that it learns through a predetermined data and generates the situation information using at least one machine learning model for predicting situation information from input integrated information.

9. In paragraph 8, The above rule-based model A customized service providing device characterized in that the situation information is determined based on threshold criteria and importance determined for each of the above element information.

10. In paragraph 8, The above machine learning model A customized service providing device characterized in that the model is learned based on the above integrated information, conditions used in the rule-based model, usual routine information about the user's repetitive routine, and previously generated situation information.

11. In paragraph 8, The above situational awareness unit Generating the situation information using at least one of the rule-based model and the machine learning model, Compare the situation information generated through the above rule-based model with the situation information generated through the above machine learning model, A customized service providing device characterized in that, through the above comparison, the result of the machine learning model is used for different information among the situation information, and if the probability value of the result of the machine learning model is below a threshold value, the result of the rule-based model is used.

12. In paragraph 8, The above situational awareness unit Generating the situation information using at least one of the rule-based model and the machine learning model, A customized service providing device characterized in that at least some of the detailed element information constituting the above situation information uses the results of the rule-based model, and the remainder of the detailed element information constituting the above situation information uses the results of the machine learning model.

13. In paragraph 1, The above service provider A customized service providing device characterized by providing at least one of a service for outputting a video based on the above situation information, a service for outputting a voice based on the above situation information, and a service for making an emergency call to a preset device based on the above situation information.

14. A method for providing customized services using robots, A moving step of controlling a moving device to move to a place where the user can be recognized; A collection step of collecting photographic information related to the user through a visual sensor; An analysis step of inputting the information collected in the above collection step into at least one analysis model and deriving element information for each analysis model; A situation recognition step of generating integrated information by integrating the element information of each derived analysis model, and generating situation information by recognizing the current situation of the user based on the integrated information; and A method for providing a customized service, characterized by including a service providing step of providing a customized service to the user based on the above situation information.

15. In paragraph 14, The above collection steps are A method for providing a customized service, characterized in that it further collects at least one piece of information from among time information for the time at which the above-mentioned shooting information is collected, robot location information generated by recognizing the location of the place where the robot is located through a location sensor, recording information generated by recognizing the user's voice or sounds occurring in the user's surroundings through a sound sensor, and device information generated from another smart device.

16. In paragraph 14, The above device information is Information generated by the smart device by detecting the location of the smart device, A method for providing a customized service, characterized in that the robot controls the moving part to move to a location where the user is located based on the device information.

17. In paragraph 14, The above device information is Information generated by the smart device by detecting the movement of the smart device, A method for providing a customized service, characterized in that the above situation recognition step generates the situation information based on the element information and the device information.

18. In paragraph 14, The above analysis model is A method for providing a customized service, characterized in that it includes at least one of a space classification model for determining the type of space in an image included in the above shooting information, an object detection model for determining the location and type of an object in an image included in the above shooting information, and a behavior recognition model for determining the posture of the user in an image included in the above shooting information.

19. In paragraph 14, The above analysis model is Including a face detection model for detecting the user's face in the image included in the above shooting information and determining the location of the face, A method for providing a customized service using a robot, characterized in that it further includes at least one of an identity recognition model for recognizing the identity of the user from a facial image generated based on the determined location of the face, an expression recognition model for recognizing the facial expression of the user from a facial image generated based on the determined location of the face, and a facial direction recognition model for recognizing the direction of the face from a facial image generated based on the determined location of the face.

20. In paragraph 14, The above situation awareness stage is A rule-based model that determines situation information based on whether element information from each of the above analysis models satisfies a predetermined condition. A method for providing a customized service, characterized in that it is a step of generating the situation information using at least one machine learning model for predicting situation information from input integrated information and learning through a predetermined amount of data.

Citation Information

Patent Citations

  • Emotion estimation system and emotion estimation model generation system

    JP2018072876A

  • A system for user-robot interaction, and information processing method for the same

    KR1020170105768A

  • Foldable electronic device and method for driving speech recognition funtion in the same

    KR1020210046475A

  • Motor for compressor, method for manufactuaring the motor for compressor, and refrigerator including the motor for compressor

    KR1020240058377A

  • System and method for wireless connection through beam-forming of wireless LAN formed by mobile communication link

    KR102237278B1