Apparatus and method for providing augmented reality content to resolve uncertainty of artificial intelligence inference-based computer vision

Abstract

Embodiments disclosed in the present specification relate to an apparatus and a method for providing augmented reality content. The apparatus for providing augmented reality content according to one embodiment comprises: a sensor unit for acquiring an image of an object in the real world; and a control unit for inputting the image into an artificial intelligence model to detect the object included in the image, and providing a hierarchy-based interaction for the detected object in a case of a value equal to or greater than a pre-configured reference threshold value according to a confidence score for the detected object.

Description

Device and method for providing augmented reality content to resolve computer vision uncertainty based on artificial intelligence inference.

[0001] The embodiments disclosed in this specification relate to an augmented reality content providing device and method for resolving computer vision uncertainty based on artificial intelligence inference, and more particularly, to an augmented reality content providing device and method capable of resolving inconvenience caused by errors due to uncertainty in artificial intelligence inference based computer vision interaction applied to augmented reality content such as augmented reality games.

[0002] This study was conducted as a result of the "Artificial Intelligence Graduate School Support (Seoul National University)" project (IITP-2021-0-01343) funded by the Ministry of Science and ICT and the National IT Industry Promotion Agency (NIPA) under the Information and Communications Broadcasting Innovation Talent Development Project, and the "Hyper-Realistic Continuous Hybrid Telepresence Platform" project (NRF-2022R1A2C3008495) funded by the Ministry of Science and ICT and the National Research Foundation of Korea under the Individual Basic Research Project.

[0003] Augmented Reality (AR) is a field of Virtual Reality (VR) that synthesizes computer-generated virtual objects or information into an existing environment (i.e. the real world) to make them appear as if they were objects in the original environment.

[0004] Augmented reality can be applied to a wide range of fields. For example, in the medical field, human dissections can be performed while wearing AR glasses, and in interior design, designers can use AR glasses to view the final appearance of the space they are working on. In the gaming industry, as described in Korean Patent Publication No. 10-2024-0039310, AR-based game content is being developed, where game characters appear in real-world spaces and interact with the real-world environment to facilitate gameplay.

[0005] These AR-based content utilize AI-based computer vision technology to recognize real-world objects and provide user interaction based on this recognition. However, object recognition using AI can be prone to errors due to the nature of AI inference, such as misrecognizing objects that don't actually exist or recognizing them differently from actual objects. This can lead to incorrect user interactions, resulting in discomfort or reduced user satisfaction with the content.

[0006] Therefore, a different level of technology is needed to compensate for errors that may arise due to the uncertainty of artificial intelligence inference or to alleviate user inconvenience by determining the type of interaction to be provided to the user differently.

[0007] Meanwhile, the background technology described above is technical information that the inventor possessed for the purpose of deriving the present invention or acquired during the process of deriving the present invention, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the application for the present invention.

[0008] An embodiment disclosed in this specification aims to present an augmented reality content providing device and method capable of reducing the failure rate of an interaction by providing a hierarchy-based interaction capable of correcting errors that may occur due to uncertainty in artificial intelligence inference.

[0009] In addition, one embodiment disclosed in the present specification aims to present an augmented reality content providing device and method that provides a user with a level of confidence in object detection, thereby allowing the user to understand whether object detection was performed correctly when an error in interaction occurs.

[0010] In addition, one embodiment disclosed in this specification aims to present an augmented reality content providing device and method that can reflect the intention of a user who prefers interaction by enabling threshold adjustment according to the intention of the user.

[0011] As a technical means for achieving the above-described technical task, according to one embodiment, an augmented reality content providing device is disclosed, including a sensor unit that acquires an image of an object in the real world, and a control unit that inputs the image into an artificial intelligence model to detect an object included in the image, and provides a hierarchy-based interaction for the detected object when the reliability score for the detected object is higher than a preset reference threshold.

[0012] According to another embodiment, a method for providing augmented reality content is disclosed, which is performed by an augmented reality content providing device, and includes the steps of obtaining an image including an object in the real world, inputting the image into an artificial intelligence model to detect an object included in the image, and providing a hierarchy-based interaction for the detected object when a reliability score for the detected object is equal to or higher than a preset reference threshold.

[0013] According to another embodiment, a computer program stored in a recording medium is disclosed for performing an augmented reality content providing method, which is performed by an augmented reality content providing device, and which comprises the steps of obtaining an image including an object in the real world, inputting the image into an artificial intelligence model to detect an object included in the image, and providing a hierarchy-based interaction for the detected object when a reliability score for the detected object is equal to or higher than a preset reference threshold.

[0014] According to another embodiment, a computer-readable recording medium having recorded thereon a program for performing an augmented reality content providing method is disclosed, wherein the augmented reality content providing method comprises the steps of: obtaining an image including an object in the real world; inputting the image into an artificial intelligence model to detect an object included in the image; and providing a hierarchy-based interaction for the detected object when a reliability score for the detected object is equal to or higher than a preset reference threshold.

[0015] According to any one of the aforementioned problem solving means, the augmented reality content providing device according to one embodiment can successfully hide minor errors in ambiguous expressions when reliability is low by providing hierarchy-based interaction, and can reduce the user's perceptual error rate, thereby reducing the negative experience of an uncertain system.

[0016] In addition, according to any one of the aforementioned problem solving means, the augmented reality content providing device according to one embodiment can provide the user with reliability regarding object detection, thereby enabling the user to understand whether object detection was performed correctly when an error in interaction occurs.

[0017] In addition, according to any one of the aforementioned problem solving means, the augmented reality content device can adjust the threshold according to the user's intention even when it continues to fall below a specific, specific level of reliability, thereby reflecting the intention of the user who prefers interaction and increasing interest in game play.

[0018] The effects that can be obtained from the disclosed embodiments are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the disclosed embodiments belong from the description below.

[0019] FIG. 1 is a schematic diagram illustrating an augmented reality content provision system according to one embodiment.

[0020] FIG. 2 is a drawing for explaining augmented reality content provided by an augmented reality providing device according to one embodiment.

[0021] Figure 3 is a block diagram illustrating the configuration of an augmented reality providing device according to one embodiment.

[0022] FIG. 4 is a drawing for explaining the operation of an augmented reality providing device according to one embodiment.

[0023] Figures 5 to 7 are flowcharts for explaining a method for providing augmented reality content according to one embodiment.

[0024] Below, various embodiments are described in detail with reference to the attached drawings. The embodiments described below may be modified and implemented in various different forms. To more clearly explain the features of the embodiments, detailed descriptions of matters widely known to those skilled in the art to which the embodiments pertain below have been omitted. In addition, parts of the drawings that are not related to the description of the embodiments have been omitted, and similar parts have been designated with similar drawing reference numerals throughout the specification.

[0025] Throughout the specification, when a component is said to be "connected" to another component, this includes not only the "direct connection" but also the "connection with other components in between." Furthermore, when a component is said to "include" another component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated.

[0026] The embodiments will be described in detail with reference to the attached drawings below.

[0027] However, before explaining this, the meanings of terms used in this specification are first defined.

[0028] 'Augmented reality' refers to adding virtual information such as graphics or sound to the real environment, and 'augmented reality content' refers to content based on augmented reality, and more specifically, refers to content to which augmented reality technology is applied.

[0029] "Object Detection" can be used interchangeably with "Object Detection," and can refer to detecting the location and type of an object within an image. Here, "object" can refer to an object excluding the background. In this specification, object recognition can be performed by an artificial intelligence model, and the artificial intelligence model can be a Deep Neural Network (DNN) or Convolutional Neural Network (CNN)-based artificial intelligence model.

[0030] A "bounding box" is the smallest rectangle that encloses an entire object. Typically, the location of a recognized object is represented by a bounding box, which can be represented by the box's center point, its width and height, or a pair of minimum and maximum x and y values.

[0031] "Confidence" or "confidence score" refers to the probability that an object exists within a bounding box. Depending on the implementation, the confidence score can be calculated simply as the probability that an object exists, the probability that an object belongs to a specific class, or the product of the probability that an object belongs to a specific class and the Intersection Over Union (IoU). During the learning phase, the confidence score can also be calculated as the product of the probability that an object exists and the IoU.

[0032] 'IoU' (Intersection Over Union) refers to the degree of overlap between the ground truth and the predicted values ​​of an AI model. In other words, it represents the ratio of the intersection of the actual and predicted bounding boxes to the union. For example, if the ground truth and predicted values ​​are nearly identical, IoU can have a value close to 1.

[0033] The embodiments will be described in detail with reference to the attached drawings below.

[0034] FIG. 1 is a block diagram illustrating an augmented reality content providing system according to one embodiment, and referring to FIG. 1, an augmented reality content providing system (10) according to one embodiment includes an augmented reality content providing device (100), and the augmented reality content providing device (100) according to one embodiment adds virtual information to the real world using augmented reality technology, and can provide a user with content that interacts with a real world object (30), i.e., augmented reality content. The augmented reality content may be, for example, game content in which virtual characters and items are superimposed on the real world and game play is performed through virtual characters, or route guidance content that displays the names of buildings and the route to the destination from the current location.

[0035] At this time, the added virtual information may be images, sounds, etc. implemented in digital graphics, and in order to display the virtual information by overlaying it on the real world, the augmented reality content providing device (100) may include a smartphone, tablet, smart glasses, laptop, or head-mounted display on which augmented reality can be displayed.

[0036] The augmented reality content providing device (100) can obtain an image including an object to be interacted with through a sensor capable of recognizing an object to be interacted with, such as a camera sensor, and can generate augmented reality content based on the obtained image.

[0037] FIG. 2 is a diagram for explaining augmented reality content provided by an augmented reality providing device according to one embodiment. For example, the augmented reality content providing device (100) according to one embodiment may be a smartphone (33) including a camera sensor as illustrated in FIG. 2, and the augmented reality content provided by the augmented reality content providing device (100) may be augmented reality-based game content in which an animal character interacts with objects in the real world.

[0038] In other words, the augmented reality content according to one embodiment may be game content in which a virtual game character interacts with objects in the real world. For example, the augmented reality content according to one embodiment may be a game played on a portable smartphone as illustrated in FIG. 2, which may be an augmented reality-based pet breeding simulation game. Specifically, the augmented reality content according to one embodiment may be performed by manipulating an actual object (31) on a flat surface, such as a desk (50), captured by the rear camera of the smartphone (33) as illustrated in FIG. 2, and may be raised by satisfying the needs of a virtual pet (40) displayed on the screen and acquiring experience points (EXP) to raise the pet by raising the pet's level.

[0039] To this end, the augmented reality content providing device (100) needs to perform the task of capturing an object in the real world, i.e., a banana (31), through a camera sensor to obtain an image containing the banana, and detecting (recognizing) the object (32) within the image, and the augmented reality content providing device (100) can detect the object within the obtained image using computer vision technology.

[0040] The augmented reality providing device (100) can input the acquired image into an artificial intelligence model to obtain information about an object in the input image, and output an interaction for the detected object based on the obtained information. For example, the information about the object may include the location of the object in the image, information about the class to which the object is classified, and a confidence score. As an example, the artificial intelligence model used for object recognition may be a CNN-based SSD (Single Shot Multibox Detector), but is not limited thereto.

[0041] Meanwhile, since the information about objects acquired by the AI ​​model is not the actual object, but the result of the AI ​​model's analysis of the input image and inference, the accuracy of object detection varies depending on the performance of the AI ​​model, and errors in inference that differ from reality may occur. For example, errors such as inferring the presence of an object when it is not contained in the image (false detection error), inferring the absence of an object when it is contained in the image (missing detection error), incorrectly judging the location of an object (location error), or incorrectly judging the class, or type, of an object (misclassification) may occur.

[0042] The four errors described above have the problem of providing incorrect state transitions or incorrect interactions in augmented reality content with various states, such as games.

[0043] To address these errors, the augmented reality content providing device (100) can set multiple reliability levels based on reliability scores and provide different interactions depending on the reliability level to which the reliability score included in the detection results belongs. Details regarding the method for setting reliability levels and the interactions provided at each reliability level will be described below.

[0044] Such an augmented reality content providing device (100) may be implemented as an electronic terminal with an application installed, or as a server-client system. When the augmented reality content providing device (100) is implemented as a server-client system, it may include an electronic terminal with a client installed for interaction with a user.

[0045] At this time, the electronic terminal may be implemented as a computer, portable terminal, television, wearable device, etc. that may include an interface that allows interaction with a user. Here, the computer includes, for example, a notebook, desktop, laptop, etc. equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, such as a PCS (Personal Communication System), a PDC (Personal Digital Cellular), a PHS (Personal Handyphone System), a PDA (Personal Digital Assistant), a GSM (Global System for Mobile communications), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet), a smart phone, a mobile WiMAX (Mobile Worldwide Interoperability for Microwave Access), etc., and may include all kinds of handheld-based wireless communication devices. Additionally, television may include Internet Protocol Television (IPTV), Internet Television, terrestrial TV, cable TV, etc.

[0046] And the server can be implemented as an electronic terminal with a client installed for interaction with a user and a computing device capable of communicating through a network, and can include a storage device capable of storing data or can store data through a third-party server.

[0047] FIG. 3 is a block diagram illustrating the configuration of an augmented reality content providing device according to one embodiment.

[0048] Referring to FIG. 3, an augmented reality content providing device (100) according to one embodiment may include a memory (110), a control unit (120), a sensor unit (130), and an input / output unit (140).

[0049] The memory (110) may be configured with various types of memory, and data and programs necessary for providing object recognition and augmented reality content may be installed and stored therein. For example, the memory (110) may store a data set for training an artificial intelligence model. As an example, the data set may be a COCO data set. In this case, the data set may include images and category information for objects included in the images.

[0050] Meanwhile, classes of hierarchically classified objects may be stored in the memory (110). In other words, classes of objects are classified based on hierarchy. For example, the hierarchy of objects may be a high hierarchy, a middle hierarchy, and a low hierarchy. The high hierarchy may be a major classification for a specific object, the middle hierarchy may be a medium classification, and the low hierarchy may be an object name. When storing a class for a specific object, the low hierarchy-middle hierarchy-high hierarchy may be stored together. As the class of an object goes to a lower hierarchy, that is, a lower hierarchy, the types of target objects included in the class become fewer and more specific, and the scope of the class becomes narrower. However, as the class goes to a higher hierarchy, the types of target objects included in the class of an object become more and more abstract, and the scope of the class may become wider.

[0051] For example, an object with a lower hierarchy of bananas may have a middle hierarchy of bananas, a middle hierarchy of fruits, and a higher hierarchy of food, and may be stored in memory (110) in the form of banana-fruit-food. Similarly, a donut may be stored with sweets and food, and broccoli and carrots may be stored with vegetables and food. Here, for the upper hierarchy of food, sweets, fruits, and vegetables may be the middle hierarchy, and the lower hierarchy of fruits may be bananas, apples, and oranges, and the lower hierarchy of vegetables may be broccoli and carrots.

[0052] Meanwhile, multiple interaction levels can be stored in the memory (110). The interaction levels can be determined based on a reliability score, set based on multiple thresholds, and different types of interactions can be provided depending on the interaction level.

[0053] For example, assuming there are three interaction levels, the first interaction level may be the lowest level, offering the fewest types of interactions, while the third interaction level may be the highest level, offering the most types of interactions. Further details regarding the types of interactions offered by each interaction level will be provided below.

[0054] Meanwhile, interaction levels may be stored in memory (110) by matching them to the hierarchy of object classes. For example, a first interaction level may be stored by matching them to a class in a higher hierarchy, a second interaction level may be stored by matching them to a class in a middle hierarchy, and a third interaction level may be stored by matching them to a class in a lower hierarchy.

[0055] In addition, the memory (110) may store a plurality of thresholds, and may include a reference threshold that serves as a criterion for providing hierarchical-based interaction, and a plurality of thresholds that determine an interaction level. For example, the plurality of thresholds may be first to fourth thresholds, with the first threshold being the lowest threshold, which is the threshold with the lowest magnitude, the second threshold being the reference threshold, and the second to fourth thresholds being thresholds that serve as criteria for determining an interaction level.

[0056] Additionally, according to an embodiment, the memory (110) may store a plurality of thresholds, and a plurality of reliability levels set based on the thresholds may also be stored. The plurality of thresholds may be minimum values ​​that determine the range of reliability scores belonging to each reliability level. For example, the first threshold may be the minimum value of the reliability level "low", and the second threshold may be the minimum value of the reliability level "medium". In other words, the reliability level "low" may be a reliability level assigned to an object having a reliability score greater than or equal to the first threshold and less than the second threshold.

[0057] Some confidence levels can be stored in association with interaction levels. For example, a confidence level of "medium" can be associated with a first interaction level, a confidence level of "high" can be associated with a second interaction level, and a confidence level of "highest" can be associated with a second interaction level. Here, the "medium" confidence level, which is the lowest level at which hierarchical interaction is provided, can be set as the reference level for providing hierarchical interaction, and the second threshold, which is the minimum value of the reference level, can be stored as the reference threshold.

[0058] Meanwhile, an artificial intelligence model used for object detection can be stored in the form of a program in the memory (110), and a program required to provide augmented reality content, such as an augmented reality-based game, can be stored.

[0059] The control unit (120) is a configuration including at least one processor, such as a CPU, a GPU, etc., and can perform the augmented reality content providing method presented below by executing a program stored in the memory (110). The process of the control unit (120) performing the augmented reality content providing method according to one embodiment will be described in detail below with reference to other drawings.

[0060] The sensor unit (130) can acquire objects, backgrounds, etc. of the real world. The sensor unit (130) can capture an image or video of an environment or object intended by the user. For example, the sensor unit (130) can be a camera sensor attached to the front or back of a smartphone, or a camera module mounted on AR glasses.

[0061] The input / output unit (140) can receive input from a user, or display information such as interaction and inference reliability for augmented reality content and objects detected by the user. For example, if the augmented reality content providing device (100) incorrectly recognizes the location of an object or recognizes an object different from the actual object, such as an apple as a banana, the device can provide an interface for receiving feedback from the user through the input / output unit (140), and can receive feedback input from the user. The input / output unit (140) can include various types of input devices (e.g., a keyboard, a touch screen, a camera, etc.) for receiving input from the user.

[0062] Additionally, the input / output unit (140) may include a display for displaying augmented reality content, which may be provided in the form of a camera sensor of the sensor unit (130) and AR glasses, for example. Additionally, an output device such as a speaker may be included to provide sound included in the augmented reality content.

[0063] Meanwhile, although not shown, the communication unit (not shown) can perform wired or wireless communication with other devices or networks. For example, the communication unit (not shown) can receive data necessary for training an artificial intelligence model or receive patch files for updating an augmented reality content provider program.

[0064] To this end, the communication unit (not shown) may include a communication module that supports at least one of various wired and wireless communication methods, and the communication module may be implemented in the form of a chipset. The wireless communication supported by the communication unit (130) may be, for example, WiFi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra-Wide Band), or NFC (Near Field Communication).

[0065] Below, a detailed description will be given of an augmented reality content provision process according to one embodiment, performed by the control unit (120) by executing a program stored in the memory (110). The processes described below are performed by the control unit (120) by executing a program stored in the memory (110), unless otherwise specifically stated.

[0066] FIG. 4 is a diagram illustrating the operation of an augmented reality content providing device according to one embodiment. For convenience of explanation, the augmented reality content providing device according to one embodiment is limited to providing interaction with a single object, and the operation of the augmented reality content providing device according to one embodiment will be described based on an augmented reality-based pet breeding simulation game played on a portable smartphone as illustrated in FIG. 2.

[0067] Referring to FIG. 4, the control unit (120) receives an image captured by the sensor unit (130), inputs the acquired image into an artificial intelligence model (410) in the form of frame images, and detects objects such as a hand (412) or an object (413) that is the target of interaction included in the image to obtain information about the objects. In addition, the control unit (120) can generate an interaction (420) based on information about the hand (412) or the object (413) that is the target of interaction, and provide the generated interaction to the user.

[0068] The artificial intelligence model (410) can detect objects included in each frame image that constitutes the video. For example, in the case of the augmented reality content illustrated in FIG. 2, the objects included in the frame images may be a plane (411) such as a desk (50), a hand (412), and objects in the real world, and some of the objects in the real world may be objects (413) that are targets of interaction. At this time, the objects (413) that are targets of interaction may be set differently depending on the augmented reality content. For example, in the case of the augmented reality-based game content of FIG. 2, the objects that are targets of interaction may be bananas, donuts, carrots, apples, oranges, cups, etc.

[0069] The artificial intelligence model (410) can output information about the detected object. At this time, the information about the object may include, for example, information about the class of the object including the object name, a confidence score, and a bounding box indicating the location of the object.

[0070] The control unit (120) can generate an interaction (420) based on information about the detected object. For example, if the detected object is a hand (412), an interaction can be generated to move the companion animal (40) of FIG. 2 to the location where the hand was detected, or if the hand (412) is detected in response to a specific request, it can be determined as a positive response to the request.

[0071] If a detected object receives information about an object (413) with which interaction is possible, an interaction can be created (420). Hereinafter, the reliability score described below refers to the reliability score for the object (413) that is the target of the interaction, and the object class refers to the object class of the object (413) that is the target of the interaction.

[0072] The control unit (120) can generate an interaction based on the reliability score for an object, as shown in Table 1 below. The generated interaction may be a hierarchy-based interaction, a request for a user to adjust a threshold, or advice.

[0073] [Table 1]

[0074]

[0075] Referring to Table 1, the threshold may be a plurality of preset values, and the lowest threshold among the plurality of thresholds may be referred to as the first threshold, and the reference threshold for determining whether an object is detected may be a second threshold having a value greater than the first threshold.

[0076] The control unit (120) can determine whether an object is detected based on whether the reliability score is greater than or equal to a reference threshold, and can determine that the object (413) that is the target of the interaction has been detected only when the reliability score is greater than or equal to the reference threshold.

[0077] Additionally, according to an embodiment, the control unit (120) may determine that an object has been detected if the reliability score of the object (412) that is the target of the interaction exceeds a second threshold value for a predetermined number of consecutive frames, for example, 10 consecutive frames.

[0078] Meanwhile, if an object is not detected for a certain period of time or a certain number of frames, it may be the case that the object does not actually exist in the image, but it may also be the case that the artificial intelligence model (410) failed to detect the object (missing error). Accordingly, if a reliability score lower than the second threshold is obtained, the control unit (120) may perform error correction (431) during detection, and in this case, error correction (431) during detection may include threshold adjustment that lowers the second threshold to a certain level.

[0079] Specifically, when the reliability score is less than the second threshold, which is a reference threshold, for a certain period of time or a certain frame, the control unit (120) may provide a threshold control request message to the user or provide a tip that can improve object detection accuracy while having the pet make a gesture of looking at an arbitrary location.

[0080] For example, the control unit (120) can display a message such as “Nothing is detected. If there is an object to be detected, please show your hand” through the input / output unit (140), and then, if the user’s hand (412) is detected from the input frame image, the control unit (120) can perform post-processing (414) to lower the second threshold value, thereby increasing the sensitivity of object detection, thereby detecting the object (413) that is the target of the interaction.

[0081] At this time, the control unit (120) can change the second threshold to be the same as the first threshold, which is the lowest threshold, when executing threshold adjustment, or can reduce the second threshold by a preset amount, and after a certain period of time, can change the second threshold back to the existing second threshold value.

[0082] Additionally, as another example, the control unit (120) may, through the input / output unit (140), notify that an object is not detected and provide a message such as “Please move the camera slowly.”

[0083] According to an embodiment, the control unit (120) may provide a tip for improving object detection accuracy for a confidence score below a first threshold output from the artificial intelligence model (410), and may provide a message for confirming whether threshold adjustment is to be executed for a confidence score between the first threshold and the second threshold.

[0084] An augmented reality providing device (100) according to one embodiment can improve the reliability of a detection result by providing a message as advice that can increase accuracy when a reliability score lower than a first threshold is obtained from an artificial intelligence model (410), and can adjust the detection sensitivity according to the user's intention by checking whether threshold adjustment is executed, thereby reflecting the user's intention to increase the frequency of interaction, thereby increasing the user's satisfaction with the content.

[0085] Meanwhile, referring back to Table 1, the control unit (120) can generate (420) an interaction for an object (413) that is the target of the interaction based on information about the object (413) that is the target of the interaction obtained from the artificial intelligence model, and provide the generated interaction to the user. However, as described above, the reliability of the inference of the artificial intelligence model (410) that performs object detection is not 100%, and therefore, errors due to uncertainty in inference, such as location errors and misclassifications, may be included in the information about the object (413) that is the target of the interaction, and thus, an incorrect interaction may be generated.

[0086] For example, a real object may be an apple, but it may be detected as a cup, resulting in an interaction where a virtual character washes the apple, or the location of the object may be inferred incorrectly, resulting in an interaction where a virtual character eats an apple in a location where there is no apple.

[0087] The control unit (120) can reduce the generation of incorrect interactions by applying expression error correction (432), except in cases where the reliability score is higher than the fourth threshold value, which is the highest. Specifically, expression error correction (432) can be performed by generating a hierarchy-based interaction for the detected object.

[0088] Specifically, the control unit (120) can determine the level of one interaction among multiple interaction levels stored in the memory (110) based on the reliability score as shown in Table 1, and can generate an interaction corresponding to the determined interaction level. The interaction level is divided based on multiple different thresholds based on the reliability score, and as shown in Table 1, a value higher than or equal to the second threshold and lower than the third threshold can be a first interaction level, a value higher than or equal to the third threshold and lower than the fourth threshold can be a second interaction level, and a value higher than or equal to the fourth threshold can be a third interaction level. In this case, the third threshold can have a value greater than the second threshold, and the fourth threshold can have a value greater than the third threshold.

[0089] Additionally, as described above, the interaction level can be stored in accordance with the class hierarchy of the object, and the control unit (120) can generate and provide relatively more types of interactions for a higher interaction level, such as a third interaction level, than for a lower interaction level, such as a first or second interaction level.

[0090] For example, if the detected object is an apple and a reliability score greater than or equal to a second threshold and less than or equal to a third threshold is measured, the control unit (120) can determine the interaction level for the apple as the first interaction level, and the control unit (120) can provide a gesture of eating 'food', which is a higher hierarchical class of 'apple'.

[0091] If the detected object is an apple and a reliability score greater than or equal to the third threshold and less than or equal to the fourth threshold is measured, the control unit (120) can determine the interaction level for the apple as the second interaction level, and the control unit (120) can generate a message indicating that 'fruit', which is a middle hierarchical class of 'apple', is good and an eating gesture as an interaction and provide the message to the user. In addition, if the detected object is an apple and a reliability score greater than or equal to the fourth threshold is measured, the interaction level for the apple can be determined as the third interaction level, and the control unit (120) can generate a message indicating that 'apple', which is a lower hierarchical class of 'apple', is being eaten and a gesture indicating that a pet is eating an apple icon as an interaction and provide the message to the user.

[0092] An augmented reality content providing device (100) according to one embodiment can increase the ambiguity of interactions by providing hierarchical interactions. This can successfully conceal minor errors within ambiguous expressions when reliability is low, and can reduce the user's perceptual error rate, thereby mitigating negative experiences associated with uncertain systems.

[0093] In summary of the above, the augmented reality content device (100) according to one embodiment provides hierarchy-based interaction only when the threshold value is above a preset threshold value, but when it continues to be below the threshold value, the threshold value can be adjusted according to the user's intention, thereby reflecting the intention of the user who prefers interaction and increasing interest in game play.

[0094] Meanwhile, the control unit (120) may provide a reliability notification when providing an interaction. For example, the control unit (120) may provide a reliability notification UI that can display reliability through the input / output unit (140), and may display reliability according to the provided interaction level on the reliability notification UI.

[0095] For example, the control unit (120) may include a reliability notification UI on one side of the game screen. The reliability notification UI may be a window displaying a message or an icon display box displaying an icon determined based on the reliability.

[0096] For example, if the control unit (120) determines that the interaction level for the banana is the third interaction level because a reliability level higher than or equal to the fourth threshold is obtained for the banana, the control unit (120) may output an icon corresponding to 'high reliability' or a message saying 'high reliability' on the reliability notification UI. On the other hand, if the control unit (120) determines that the interaction level for the banana is the first interaction level because a reliability level higher than or equal to the second threshold but lower than the third threshold is obtained for the banana, the control unit (120) may display an icon corresponding to 'low reliability' or a message saying 'low reliability' on the reliability notification UI. Similarly, if the interaction level for the banana is determined to be the second interaction level, the control unit (120) may display a message saying 'medium reliability' or an icon corresponding to 'medium reliability' on the reliability notification UI.

[0097] An augmented reality content providing device (100) according to one embodiment can provide a user with a level of reliability regarding object detection, thereby enabling the user to understand whether object detection was performed correctly when an error in interaction occurs.

[0098] Meanwhile, according to another embodiment, the control unit (120) determines a reliability level based on a reliability score for an object that is the target of an interaction output from an artificial intelligence model as shown in Table 2 below, and provides an interaction to a user according to the determined reliability level. However, the interaction provided to the user may be one of a hierarchy-based interaction, a confirmation of whether to execute threshold adjustment, or provision of advice to increase the accuracy of object detection.

[0099] [Table 2]

[0100]

[0101] Referring to Table 2, reliability levels can be defined based on the first to fourth thresholds, and a total of five levels, “lowest”, “low”, “medium”, “highest”, and “highest”, can be set. The control unit (120) determines the reliability level to which the reliability score belongs, and can provide hierarchy-based interaction for reliability levels “medium”, “highest”, and “highest”, which are reliability levels to which the reliability score equal to or higher than the second threshold belongs. The reliability levels “medium”, “highest”, and “highest” can be respectively matched to the first to third interaction levels. As described above, among the reliability levels for which hierarchy-based interaction is provided, the reliability level “medium”, which is the lowest reliability level, can be a reference level for providing hierarchy-based interaction, and a second threshold, which is the minimum value among the range of reliability scores corresponding to the reference level, can be a reference threshold.

[0102] Additionally, the control unit (120) can provide an upper-level interaction for a reliability level of “medium”, a middle-level interaction for a reliability level of “high”, and a lower-level interaction for a reliability level of “highest”.

[0103] As in one embodiment, the control unit (120) can provide lower-level interactions as the reliability level increases, and can provide relatively more diverse types of interactions compared to lower reliability levels.

[0104] The control unit (120) can generate and provide relatively more types of interactions for a higher interaction level, such as a third interaction level, than for a lower interaction level, such as a first or second interaction level.

[0105] For example, if the detected object is an apple and the confidence level is determined to be “medium,” the control unit (120) can determine the interaction level for the apple as a first interaction level matching the confidence level “medium,” and the control unit (120) can provide a gesture of eating “food,” which is a higher hierarchical class of “apple.” In addition, if the detected object is an apple and the confidence level is determined to be “high,” the control unit (120) can determine the interaction level for the apple as a second interaction level matching the confidence level “high,” and the control unit (120) can generate an interaction message indicating that “fruit,” which is a middle hierarchical class of “apple,” is good, and provide an eating gesture to the user.

[0106] In addition, if the detected object is an apple and the reliability level is determined to be “highest,” the control unit (120) can determine the interaction level for the apple as the third interaction level matching the reliability level “highest,” and can provide the user with a message that the pet is eating an “apple,” which is a lower hierarchical class of “apple,” and a gesture of the pet eating an apple icon as an interaction.

[0107] In addition, as in one embodiment, the control unit (120) can provide a reliability notification UI through the input / output unit (140), and can display a reliability corresponding to a reliability stage when providing an interaction on the reliability notification UI, thereby enabling a user to understand whether object detection was performed correctly when an error in the interaction occurs.

[0108] Meanwhile, if the reliability score acquired over a certain period of time falls within the reliability level "low," the control unit (120) may inquire of the user as to whether threshold adjustment should be performed, and may perform threshold adjustment in response to the user's response. When performing threshold adjustment, the control unit (120) may determine the interaction level for the object in question to be the first interaction level, and provide interaction of a higher hierarchy, even if the reliability score falls within the reliability level "low" for a certain period of time.

[0109] Additionally, if the reliability score acquired over a certain period of time is at the "lowest" reliability level, the control unit (120) may provide advice to the user to increase the reliability of the detection results. For example, the control unit (120) may notify the user that an object is not detected and provide the message "Please move the camera slowly" via the input / output unit (140).

[0110] In summary of the above, the augmented reality content device (100) according to another embodiment provides hierarchy-based interaction at a specific reliability level, i.e., above a reference level, but allows for threshold adjustment according to the user's intention when a reliability level below the reference level persists, thereby reflecting the intention of the user who prefers interaction and increasing interest in game play.

[0111] Furthermore, by providing hierarchical interactions for confidence levels above the baseline level, minor errors can be successfully hidden within ambiguous expressions when confidence levels are low, thereby reducing the user's perceptual error rate and thus reducing the negative experience of uncertain systems.

[0112] The term '~ unit' used in the above embodiments means a software or hardware component such as an FPGA (field programmable gate array) or an ASIC, and the '~ unit' performs certain roles. However, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the '~ unit' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0113] The functionality provided within the components and '~sub-components' may be combined into a smaller number of components and '~sub-components' or separated into additional components and '~sub-components'.

[0114] Additionally, the components and '~parts' may be implemented to regenerate one or more CPUs within the device or secure multimedia card.

[0115] FIG. 5 and FIG. 6 are flowcharts for explaining a method for providing augmented reality content according to one embodiment.

[0116] Referring to FIG. 5, the augmented reality content providing device (100) can acquire an image including objects in the real world (S510). The augmented reality content providing device (100) can capture an image using a camera sensor included in a smartphone or AR glasses, and the image can include objects in the real world projected through the camera lens. The image can be composed of multiple frame images, and each frame image can include objects in the real world.

[0117] Next, the augmented reality content providing device (100) inputs the acquired image into an artificial intelligence model to detect an object, and provides hierarchy-based interaction for the object based on the reliability score for the detected object, confirms whether threshold adjustment is executed, or provides advice to the user (S520).

[0118] For example, the augmented reality content providing device (100) can input the acquired image into an artificial intelligence model in units of frame images to detect objects included in the image that are targets of hands or interactions, and obtain information about the detected objects. The information about the detected objects may include the object's class and reliability score.

[0119] The augmented reality content providing device (100) can provide hierarchy-based interaction for an object by comparing the reliability score with a preset reference threshold, such as a second threshold, as shown in Table 1 described above, and if the reliability score is higher than the second threshold, can confirm whether threshold adjustment is to be performed or provide advice to the user.

[0120] Specifically, referring to FIG. 6, the augmented reality content providing device (100) determines whether the reliability score is equal to or greater than a second threshold, which is a reference threshold (S610), and if it is equal to or greater than the second threshold, provides a hierarchy-based interaction (S620). For example, the augmented reality content providing device (100) may determine the level of one of multiple interaction levels based on the reliability score, and generate a hierarchy-based interaction corresponding to the interaction level.

[0121] If the reliability score for the corresponding object is not high and the determined interaction level is low, i.e., determined as the first interaction level, the augmented reality content providing device (100) may provide an interaction of a higher hierarchy matching the class of the higher hierarchy of the corresponding object. On the other hand, if the reliability score for the corresponding object is medium and the determined interaction level is medium, i.e., determined as the second interaction level, the augmented reality content providing device (100) may provide an interaction of a middle hierarchy matching the class of the middle hierarchy of the corresponding object. In this case, there may be relatively more types of interactions of the middle hierarchy than interactions of the upper hierarchy.

[0122] Meanwhile, if the level of interaction determined by the reliability score for the object is high, the augmented reality content providing device (100) can provide a lower-level interaction matching the lower-level class of the object, and the types of lower-level interactions may be greater than those of the middle or lower-level interactions.

[0123] Meanwhile, the augmented reality content providing device (100) may provide a reliability notification when providing interaction. For example, the augmented reality content providing device (100) may display the reliability level according to the level of interaction provided through a reliability notification UI. For example, the reliability notification UI may be displayed on a portion of the screen where the augmented reality content is provided, or may be a window displaying a message or an area displaying an icon indicating the reliability level.

[0124] For example, when the augmented reality content providing device (100) provides an interaction corresponding to the first interaction level, it may display a message saying 'low reliability' or an icon corresponding to 'low reliability' on the reliability notification UI. Similarly, when the augmented reality content providing device (100) provides an interaction corresponding to the second interaction level, it may display a message saying 'medium reliability' or an icon corresponding to 'medium reliability' on the reliability notification UI, and when the interaction corresponding to the third interaction level, it may display a message saying 'high reliability' or an icon corresponding to 'high reliability' on the reliability notification UI. The augmented reality content providing device (100) according to one embodiment may provide the user with a level of reliability for object detection, thereby allowing the user to understand whether object detection was performed correctly when an error in the interaction occurs.

[0125] Meanwhile, if the reliability score is less than the second threshold, which is the reference threshold, the augmented reality content providing device (100) determines whether the reliability score is equal to or greater than the lowest threshold for a predetermined period of time (S630), and if it is equal to or greater than the lowest threshold for a predetermined period of time, checks whether the user wants to adjust the threshold (S650), and if the threshold is adjusted, the second threshold can be adjusted to the first threshold, which is the lowest threshold (S660), and as the threshold is adjusted, the augmented reality content providing device (100) can create a hierarchy-based interaction for the corresponding object.

[0126] Meanwhile, if the reliability score is not higher than the lowest threshold for a predetermined period of time, it is determined whether the reliability score is lower than the lowest threshold for a predetermined period of time (S640), and if the reliability score is lower than the lowest threshold for a predetermined period of time, the augmented reality content providing device (100) can provide the user with advice to increase the reliability of object detection (S670).

[0127] As described above, the augmented reality content method according to one embodiment provides hierarchical interaction only when the confidence score is above a reference threshold, but allows the threshold to be adjusted according to the user's intention if the confidence score continues to be below the reference threshold for a certain period of time, thereby reflecting the user's preference for interaction and increasing interest in game play. In addition, by providing hierarchical interaction, minor errors can be successfully hidden within ambiguous expressions when the confidence score is low, thereby reducing the user's perceptual error rate and thus reducing the negative experience of an uncertain system.

[0128] Meanwhile, FIG. 7 is a flowchart illustrating a method for providing augmented reality content according to another embodiment. FIG. 7 illustrates a process for performing step S520 of FIG. 5 in a different manner than FIG. 6.

[0129] Referring to FIGS. 5 and 7, the augmented reality content providing device (100) inputs the image acquired in step S510 into an artificial intelligence model to detect an object, and provides hierarchy-based interaction for the object according to the reliability score for the detected object, and can check whether threshold adjustment is executed or provide advice to the user (S520).

[0130] Specifically, the augmented reality content providing device (100) can determine one of a plurality of reliability levels based on the reliability score (S710). As in FIGS. 5 and 6, the augmented reality content providing device (100) inputs an image obtained by photographing the real world into an artificial intelligence model, and inputs the image in units of frame images constituting the image into the artificial intelligence model to detect a hand included in the image or an object to be interacted with, and can include information about the object, including the class of the detected object and the reliability score. The augmented reality content providing device (100) can compare the obtained reliability score with a plurality of thresholds and determine one of the plurality of reliability levels as a reliability level that serves as a reference for generating an interaction.

[0131] Next, the augmented reality content providing device (100) determines whether the determined reliability level is higher than the reference level. For example, the reference level may be the reliability level “medium” with the second threshold as the minimum reliability score (S720). If the determined reliability level is the reference level, the reliability level “medium,” a hierarchy-based interaction may be provided (S760). The augmented reality content providing device (100) may provide the hierarchy-based interaction for the reliability levels “medium,” “high,” and “highest.”

[0132] Specifically, the augmented reality content providing device (100) can provide a higher-level interaction for a reliability level of “medium,” a middle-level interaction for a reliability level of “high,” and a lower-level interaction for a reliability level of “highest.” As in one embodiment, the control unit (120) can provide a lower-level interaction as the reliability level increases, thereby providing various types of interactions to the user.

[0133] In addition, the augmented reality content providing device (100) may provide reliability through a reliability notification UI when providing an interaction corresponding to a reliability level. For example, when providing an interaction corresponding to an intermediate reliability level, a message such as “low reliability” or an icon corresponding to “low reliability” may be displayed on the reliability notification UI. Similarly, when providing an interaction corresponding to an intermediate reliability level, a message such as “medium reliability” or an icon corresponding to “medium reliability” may be displayed on the reliability notification UI, and when providing an interaction corresponding to the highest reliability level, a message such as “high reliability” or an icon corresponding to “high reliability” may be displayed on the reliability notification UI.

[0134] Meanwhile, if the reliability score obtained over a certain period of time is at the reliability level “low,” the augmented reality content providing device (100) inquires the user about whether to execute threshold adjustment (S740), and when executing threshold adjustment, the augmented reality content providing device (100) can provide an interaction of a higher hierarchy corresponding to the reliability level “medium” even if the reliability score for a certain period of time is at the reliability level below (S750).

[0135] In addition, if the reliability score acquired over a certain period of time is lower than the reliability level “medium” but is not “low”, it is checked whether the reliability score acquired over a certain period of time is the reliability level “lowest” (S770), and if the reliability score acquired over a certain period of time is the reliability level “lowest”, the augmented reality content providing device (100) can provide advice to the user to increase the reliability of the detection result (S780). For example, it can inform the user that an object is not detected and provide a message such as “Please move the camera slowly.”

[0136] In summary, the augmented reality content method according to another embodiment provides hierarchical interaction above a certain confidence level, but allows for threshold adjustment according to the user's preference when the confidence level remains below a certain level. This allows for reflecting the user's preference for interaction and thus increasing interest in gameplay. Furthermore, by providing hierarchical interaction, minor errors can be successfully concealed within ambiguous expressions when the confidence level is low, thereby reducing the user's perceptual error rate and thus the negative experience of an uncertain system.

[0137] The method for providing augmented reality content according to an embodiment may be implemented as a computer program (or computer program product) including computer-executable instructions. The computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, an object-oriented programming language, assembly language, or machine language. Furthermore, the computer program may be recorded on a tangible computer-readable recording medium (e.g., memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD), etc.).

[0138] Accordingly, the method for providing augmented reality content according to the embodiments described with reference to FIGS. 5 to 7 can be implemented by executing the aforementioned computer program on a computing device. The computing device may include at least some of a processor, memory, a storage device, a high-speed interface connecting the memory and a high-speed expansion port, and a low-speed interface connecting the low-speed bus and the storage device. Each of these components is interconnected using various buses and may be mounted on a common motherboard or in another suitable manner.

[0139] Here, the processor can process instructions within the computing device, such as instructions stored in a memory or storage device to display graphical information for providing a graphical user interface (GUI) on an external input / output device, such as a display connected to a high-speed interface. In another embodiment, multiple processors and / or multiple buses may be utilized, as appropriate, together with multiple memories and memory types. The processor may also be implemented as a chipset comprising multiple independent analog and / or digital processors.

[0140] Memory also stores information within a computing device. For example, memory may consist of volatile memory units or a collection of volatile memory units. For another example, memory may consist of nonvolatile memory units or a collection of nonvolatile memory units. Memory may also be another form of computer-readable media, such as magnetic or optical disks.

[0141] And memory can provide a large amount of storage space for a computing device. Memory can be a computer-readable medium or a configuration including such a medium, and can include devices within a storage area network (SAN) or other configurations, such as a floppy disk drive, a hard disk drive, an optical disk drive, a tape drive, flash memory, or other similar semiconductor memory devices or arrays of devices.

[0142] The embodiments described above are provided for illustrative purposes only, and those skilled in the art will readily appreciate that the embodiments described above can be readily modified into other specific forms without altering the technical concepts or essential characteristics of the embodiments described above. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0143] The scope of protection sought through this specification is indicated by the claims described below rather than the detailed description above, and should be interpreted to include all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts.

Claims

1. A sensor unit that acquires images of objects in the real world; and An augmented reality content providing device, comprising a control unit that inputs the above image into an artificial intelligence model to detect an object included in the image, and provides a hierarchy-based interaction for the detected object when the reliability score for the detected object is higher than a preset standard threshold.

2. In paragraph 1, The above control unit, An augmented reality content providing device that checks whether to execute threshold adjustment to adjust the above-mentioned threshold or provides advice to the user when the above-mentioned reliability score is below the above-mentioned reference threshold.

3. In paragraph 2, The above control unit, An augmented reality content providing device that checks with the user whether to execute threshold adjustment when the reliability score for the detected object is below the reference threshold for a certain period of time but above the lowest threshold, and adjusts the reference threshold to the lowest threshold when threshold adjustment is executed.

4. In paragraph 3, The above control unit, An augmented reality content providing device that provides advice to a user when the confidence score for the detected object is below the lowest threshold.

5. In paragraph 1, The above control unit, An augmented reality content providing device that provides an interaction of a hierarchy corresponding to an interaction level to which the reliability score belongs among multiple interaction levels when the reliability score is greater than or equal to the reference threshold.

6. In paragraph 5, The above control unit, An augmented reality content providing device that provides lower-level interactions that include relatively many types of interactions as the interaction level increases.

7. In paragraph 5, The above control unit, An augmented reality content providing device that provides a user with a reliability notification corresponding to the interaction level to which the above reliability score belongs.

8. In paragraph 1, The above control unit, An augmented reality content providing device, which determines a reliability level to which the reliability score belongs among a plurality of reliability levels distinguished based on a plurality of thresholds, and, if the determined reliability level is higher than a reference level, provides a hierarchy-based interaction for the detected object, wherein the reference level has the reference threshold as its minimum value.

9. In paragraph 8, The above control unit, An augmented reality content providing device that checks whether a threshold adjustment is performed by a user when the reliability stage to which the reliability score belongs for a certain period of time is a reliability stage set based on the lowest threshold among the plurality of thresholds and the reference threshold, and provides an interaction of a higher level when the threshold adjustment is performed.

10. Performed by an augmented reality content providing device, A step of acquiring an image containing an object in the real world; and A method for providing augmented reality content, comprising the steps of inputting the above image into an artificial intelligence model to detect an object included in the image, and providing a hierarchy-based interaction for the detected object when the reliability score for the detected object is higher than a preset standard threshold.

11. In paragraph 10, The steps provided above are: A method for providing augmented reality content, comprising a step of checking whether a threshold adjustment is performed by a user when the reliability score for the detected object is lower than the reference threshold but higher than the lowest threshold for a certain period of time, and adjusting the threshold to the lowest threshold when adjusting the threshold.

12. In paragraph 11, The steps provided above are: A method for providing augmented reality content, comprising a step of providing advice to a user when the confidence score for the detected object is less than the lowest threshold.

13. In paragraph 10, The steps provided above are: A method for providing augmented reality content, comprising the step of providing an interaction of a hierarchy corresponding to an interaction level to which the reliability score belongs among a plurality of interaction levels, when the reliability score is greater than or equal to the reference threshold.

14. In paragraph 10, The steps provided above are: A step of determining a reliability stage to which the reliability score belongs among a plurality of reliability stages distinguished based on a plurality of thresholds; and If the above-determined reliability level is greater than or equal to the reference level, a step of providing a hierarchy-based interaction for the detected object is included. A method for providing augmented reality content, wherein the above-mentioned standard step sets the above-mentioned standard threshold as the minimum value.

15. In paragraph 14, The steps provided above are: A method for providing augmented reality content, comprising: a step of confirming whether a threshold adjustment for adjusting the reference threshold is executed by a user when the reliability stage to which the reliability score belongs for a certain period of time is a reliability stage set based on the lowest threshold among the plurality of thresholds and the reference threshold; and providing an interaction of a higher hierarchy when the threshold adjustment is executed.

16. In paragraph 15, The steps provided above are: A method for providing augmented reality content, comprising a step of providing advice to a user when the above-determined reliability level is a reliability level lower than the lowest threshold.

17. A computer program that is performed by an augmented reality content providing device and performs the method described in paragraph 10.

18. A computer-readable recording medium having recorded thereon a computer program for performing the method described in Article 10.

Images