Apparatus for providing tactility information and method for providing tactility information

Abstract

The present invention relates to an apparatus for providing tactile information and a method for providing tactile information. The method for providing tactile information, according to one embodiment of the present invention, may comprise the steps of: recognizing a target object included in an image being played back; generating, from a target object image corresponding to the target object, target characteristic information for estimating the tactile sensation of the target object; and comparing pre-stored characteristic information about a plurality of objects to the target characteristic information so as to search for an object corresponding to the target object, and extracting target tactile information about the target object from tactile information about the searched for object.

Description

Tactile information providing device and tactile information providing method

[0001] The present invention relates to a tactile information providing device and a tactile information providing method capable of recognizing an object included in an image in real time and providing a tactile sensation for the recognized object.

[0002]

[0003] As virtual reality technology advances, technologies are being researched to realistically recreate virtual reality by providing users with the five senses of the virtual reality environment, atmosphere, smell, touch, and taste, in addition to the images and sounds that represent the virtual reality.

[0004] In the past, virtual touch was conveyed to users through simple vibrations. However, this simply indicated contact with objects within a virtual reality, differing from providing virtual touch, which simulates the sensation of contact with the surface of real objects.

[0005] Furthermore, even when providing virtual tactile sensations, previously only virtual reality images with pre-stored tactile information were able to provide tactile sensations. This limitation stemmed from the inability to provide tactile sensations for arbitrary, non-predefined objects.

[0006]

[0007] The present invention provides a tactile information providing device and a tactile information providing method capable of providing a tactile sensation for objects appearing in an image.

[0008] The present invention provides a tactile information providing device and a tactile information providing method that can provide tactile information by estimating the tactile sensation based on visual information about an object, even for an object that is not defined in advance.

[0009]

[0010] A method for providing tactile information according to one embodiment of the present invention provides tactile information for reproducing a tactile sensation corresponding to an image being viewed by a user, the method including: a step of recognizing a target object included in the image being played; a step of generating target characteristic information for estimating a tactile sensation of the target object from a target object image corresponding to the target object; and a step of searching for an object corresponding to the target object by comparing the target characteristic information with characteristic information of a plurality of objects stored in advance, and extracting target tactile information of the target object from the tactile sensation information of the searched object.

[0011] Here, the step of recognizing the target object may be to store identifiers and location information corresponding to the target objects in an object tracking database, thereby tracking the location of the target object within the image.

[0012] Here, the target object image may be generated by cropping the interior of a bounding box generated corresponding to the location of the target object within the image.

[0013] Here, the step of generating the target characteristic information may be to extract at least one of the color, size, shape, movement, and distance within the image of the target object from the target object image and generate the target characteristic information.

[0014] Here, the step of extracting the target tactile information may be to compare the target characteristic information with the characteristic information of the plurality of objects stored in advance to obtain a similarity, and extract the tactile information corresponding to an object whose similarity is greater than a set value as the target tactile information of the target object.

[0015] Here, the step of extracting the target tactile information may be to input the target characteristic information into a tactile analysis model to generate target tactile information corresponding to the target object, if there is no object with the similarity greater than the set value.

[0016] Here, the tactile analysis model may generate tactile information including at least one tactile type among texture, temperature, pressure, and vibration of the target object using the characteristic information.

[0017] Here, the tactile analysis model may be learned for each object type of each predefined object, and may generate the tactile information by reflecting characteristic information according to the object type.

[0018] Here, the tactile analysis model includes each individual tactile model corresponding to at least one tactile type, and the step of extracting the target tactile information may be to obtain the similarity for each item included in the characteristic information and the target characteristic information, and if there is an item whose similarity is less than a set value, input the item whose similarity is less than the set value into the individual tactile model of the tactile type corresponding to the item whose similarity is less than the set value among each of the individual tactile models, thereby generating tactile information for the tactile type.

[0019] Here, the step of extracting the target tactile information may be to compare each identifier corresponding to the plurality of objects with an identifier corresponding to the target object, and if there is an object among the plurality of objects that has an identifier matching the identifier of the target object, compare the characteristic information of the object with the target characteristic information of the target object to obtain the similarity.

[0020] Here, the step of extracting the target tactile information may be to input the target characteristic information into a tactile analysis model to generate target tactile information corresponding to the target object, if an object having the matching identifier does not exist.

[0021] Here, the step of extracting the target tactile information may be to extract the target tactile information by further using at least one of auditory information, olfactory information, and gustatory information included in the image corresponding to the target object.

[0022] Here, the method for providing tactile information according to one embodiment of the present invention may further include a step of transmitting the target tactile information to a tactile reproduction device worn by the user when the user pointer displayed in the image is located at a point where interaction with the target object is possible.

[0023] According to one embodiment of the present invention, a computer program stored in a medium can be implemented to execute the above-described tactile information providing method in combination with hardware.

[0024] A tactile information providing device according to one embodiment of the present invention includes a processor, and provides tactile information for reproducing a tactile sensation corresponding to an image being viewed by a user, wherein the processor may perform the following actions: recognizing a target object included in the image being played; generating target characteristic information for estimating a tactile sensation of the target object from a target object image corresponding to the target object; and searching for an object corresponding to the target object by comparing the target characteristic information with characteristic information of a plurality of objects stored in advance, and extracting target tactile information of the target object from the tactile sensation information of the searched object.

[0025] Additionally, the solutions to the aforementioned problems do not enumerate all features of the present invention. The various features of the present invention, along with their corresponding advantages and effects, can be understood in more detail by referring to the specific embodiments below.

[0026]

[0027] According to a tactile information providing device and method according to one embodiment of the present invention, tactile information can be provided by estimating the tactile sensation of objects appearing in an image based on visual information. This makes it possible to provide tactile information even for arbitrary objects not defined in advance.

[0028] According to a tactile information providing device and method according to one embodiment of the present invention, tactile information previously generated based on the similarity between objects can be reused, and new tactile information can be generated using a tactile analysis model only for objects with low similarity. Through this, the amount of computation required to generate tactile information can be minimized, thereby providing tactile information about objects in real time.

[0029]

[0030] Figures 1 and 2 are schematic diagrams showing a tactile sensation providing system according to one embodiment of the present invention.

[0031] Figure 3 is a block diagram showing a tactile information providing device according to one embodiment of the present invention.

[0032] Figure 4 is an exemplary diagram showing an object appearing in an image according to one embodiment of the present invention.

[0033] Figure 5 is an exemplary diagram showing a user pointer and an object in an image according to one embodiment of the present invention.

[0034] Figure 6 is a block diagram showing a tactile information providing device according to another embodiment of the present invention.

[0035] Figure 7 is a flowchart showing a method for providing tactile information according to one embodiment of the present invention.

[0036]

[0037] Hereinafter, with reference to the attached drawings, preferred embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention. However, when describing preferred embodiments of the present invention in detail, if a detailed description of a related known function or configuration is judged to unnecessarily obscure the gist of the present invention, such detailed description will be omitted. In addition, the same reference numerals are used throughout the drawings for parts that perform similar functions and actions.

[0038] In addition, throughout the specification, when a part is said to be 'connected' to another part, this includes not only cases where it is 'directly connected', but also cases where it is 'indirectly connected' with another element in between. Furthermore, 'including' a certain component does not mean excluding other components unless specifically stated to the contrary, but rather that other components may be included. Furthermore, terms such as "part" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or a combination of hardware and software.

[0039]

[0040] Figures 1 and 2 are schematic diagrams showing a tactile sensation providing system according to one embodiment of the present invention.

[0041] Referring to FIGS. 1 and 2, a tactile sensation providing system according to one embodiment of the present invention may include an image output device (A), a tactile sensation reproduction device (B), and a tactile sensation information providing device (100).

[0042] Hereinafter, a tactile sensation providing system according to one embodiment of the present invention will be described with reference to FIGS. 1 and 2.

[0043]

[0044] The video output device (A) may be a device that displays various types of images, and may output general media broadcasts such as terrestrial or cable broadcasts, or VOD (Video on Demand) images. Depending on the embodiment, the video output device (A) may output a virtual reality image, and the virtual reality image may be an image that implements virtual reality, augmented reality, mixed reality, etc.

[0045] The image output device (A) may be a display device based on an LCD (Liquid Crystal Display), an LED (Light Emitting Diode), etc., as illustrated in FIG. 1, and, depending on the embodiment, may be implemented as an HMD (Head Mounted Display) or smart glasses worn by the user (1) on his or her head, as illustrated in FIG. 2. In addition, the image output device (A) may be a hologram device that visualizes and displays a hologram, and, depending on the embodiment, may be a device that generates a tensiable hologram that provides a tactile sensation or a sensation similar to a tactile sensation of the hologram together with the hologram visualized in space. However, the present invention is not limited thereto, and any device capable of outputting an image to the user may be used.

[0046] The image output by the image output device (A) may be a two-dimensional image or a three-dimensional image, and the image output device (A) may output an image stored in itself or output an image received through an external network through streaming, etc.

[0047] In some embodiments, the image output device (A) may provide a realistic virtual reality image to the user (1), in which case the user (1) may perceive that he or she is located within the virtual reality image rather than in a real location through the virtual reality image. For example, the image output device (A) may output a virtual reality image of riding a roller coaster, in which case the user (1) may be provided with a virtual reality image including images and sounds similar to those of the user actually riding the roller coaster through the image output device (A). Through this, the user (1) may be comfortably sitting in an actual fixed location, but may feel as if he or she is riding an actual roller coaster through the virtual reality image.

[0048] Here, the video output device (A) can give the user the impression that he or she is located within a virtual reality through sight and hearing. However, sight and hearing alone may not be enough to make the user feel as if he or she is actually located within the virtual reality. In other words, the user may require additional senses, such as touch, in addition to sight and hearing, and for this purpose, the tactile reproduction device (B) may be worn on a part of his or her body. The tactile reproduction device (B) may provide various tactile sensations corresponding to each object appearing within the virtual reality image.

[0049] However, the tactile reproduction device (B) can provide a sense of touch not only for virtual reality images, but also for general images, as illustrated in Fig. 1. In other words, it can be implemented so that a user (1) watching a general image can have a new experience by providing a sense of touch corresponding to various situations appearing in the image. For example, a user (1) can watch real-time TV using an image output device (A) and move a user pointer (HD) displayed in the image output device (A), and the tactile reproduction device (B) can provide a sense of touch corresponding to an object where the user pointer (HD) is located. Therefore, the user (1) can feel various senses of touch provided by the tactile reproduction device (B) depending on the location of the user pointer (HD) in the image output device (A).

[0050] Meanwhile, as illustrated in FIGS. 1 and 2, the tactile reproduction device (B) can be implemented as a wearable device such as a glove, clothing, necklace, or watch, but is not limited thereto and can be implemented in various forms such as a riding type or a stationary type. The tactile reproduction device (B) can generate thermal stimulation such as heat, mechanical stimulation such as vibration or pressure, and may also be capable of providing stimulation such as wind.

[0051] Depending on the embodiment, it is also possible to further include an olfactory reproduction device (not shown) that provides olfactory feedback or a gustatory reproduction device (not shown) that provides gustatory feedback. That is, in order to receive more feedback on the senses of smell and taste, the user may hold a part of the gustatory reproduction device in the mouth, or attach a part of the olfactory reproduction device to the nose or insert it inside the nose. For example, in the case of the olfactory reproduction device, a specific scent or smell may be generated according to olfactory information corresponding to the image output from the image output device (A), and in the case of the gustatory reproduction device, a specific taste such as sour or salty may be generated according to gustatory information corresponding to the image. In the case of the tactile reproduction device (B), a specific form of pressure, vibration, wind, heat, etc. may be generated according to tactile information corresponding to the image.

[0052] Additionally, although FIGS. 1 and 2 illustrate a case where the image output device (A) and the tactile reproduction device (B) are implemented independently, it is also possible to implement the image output device (A) and the tactile reproduction device (B) as a single device depending on the embodiment. For example, a device that displays a visualized hologram in space and simultaneously provides a tactile sensation or a sensation similar to the tactile sensation, such as a tangible hologram device (not shown), can be utilized. In this case, the device can generate and provide a tangible hologram in which tactile information is reflected based on the tactile information received from the tactile information providing device (100).

[0053] The tactile information providing device (100) can generate and provide tactile information for each object appearing in the image so that the tactile reproduction device (B) can reproduce the tactile sensation. The tactile reproduction device (B) can reproduce various tactile sensations that the user (1) can feel, but in order to reproduce the tactile sensation according to the interaction with each object output in the image, information about what kind of tactile sensation each object has may be required. To this end, the tactile information providing device (100) can generate and provide tactile information about the tactile sensation corresponding to each object so that the tactile sensation corresponding to the image can be reproduced.

[0054] As illustrated in FIGS. 1 and 2, the tactile information providing device (100) is configured individually with the image output device (A) or the tactile reproduction device (B) and can be connected to each other through a network. Here, the network refers to a connection structure that enables information exchange between each node, such as a terminal or a server, and may include a local area network (LAN), a wide area network (WAN), the Internet (WWW), a wired and wireless data communication network, a telephone network, a wired and wireless television communication network, etc. Examples of wireless data communication networks may include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC), LiFi, etc. However, depending on the embodiment, instead of connecting the tactile information providing device (100) through a network, it is also possible to implement it by combining it with the image output device (A) or including it within the image output device (A).

[0055] In the past, simple vibrations were used to convey tactile sensations to users, but this simply indicated that there had been contact with the object, and there was a difference from providing virtual tactile sensations that correspond to the tactile sensations of actual objects. Furthermore, even when providing virtual tactile sensations, previously, it was only possible to provide tactile sensations for images in which tactile information for each object was stored in advance to support the operation of the tactile reproduction device (B). In other words, there were limitations such as not being able to provide tactile sensations for arbitrary objects that were not defined in advance.

[0056] On the other hand, the tactile information providing device (100) according to one embodiment of the present invention can estimate the tactile sensation of each object appearing in an image based on visual information of the objects and generate corresponding tactile sensation information. In other words, it is possible to provide tactile sensation even for objects that are not predefined, and when the state of the object changes, it is possible to reflect this and provide tactile sensation.

[0057] Hereinafter, a tactile information providing device (100) according to one embodiment of the present invention will be described with reference to FIG. 3.

[0058]

[0059] Figure 3 is a block diagram showing a tactile information providing device (100) according to one embodiment of the present invention.

[0060] Referring to FIG. 3, a tactile information providing device (100) according to one embodiment of the present invention may include an object analysis unit (110), an object tracking unit (120), an object change analysis unit (130), an object meta storage unit (140), a tactile analysis unit (150), and a tactile information transmission unit (160).

[0061] As illustrated in FIG. 4, the object analysis unit (110) can recognize a target object among objects (OB1, OB2, OB3) included in the video (I) being played. Here, the object analysis unit (110) can generate a target object image corresponding to the target object, and can generate object characteristic information for estimating the touch of the corresponding target object from the target object image. That is, it is possible to generate touch information for the corresponding object by utilizing the characteristic information extracted by the object analysis unit (110). Through this, even if touch information for each object (OB1, OB2, OB3) is not included in advance in the video (I), it is possible to reproduce and provide the touch corresponding to the target object to the user (1).

[0062] Specifically, the object analysis unit (110) can recognize preset target objects using an object recognition model, and at this time, the object recognition model may be implemented using machine learning or a neural network, etc. In other words, the object recognition model may be trained to recognize target objects included in an image using learning data corresponding to each target object and create a bounding box for the target object.

[0063] Thereafter, the object analysis unit (110) can generate a target object image by cropping the interior of the bounding box generated corresponding to the location of the target object in the image (I). That is, if analysis is performed on the entire image (I), unnecessary computational amounts may increase, so the computational amounts required for object analysis can be minimized by generating a target object image corresponding to the desired target object and performing analysis limited to the target object image. For example, if the target object is an apple (OB2), the target object image can be generated by cropping the area corresponding to the apple (OB2) from the entire image (I).

[0064] Here, the object analysis unit (110) can determine the location and size of the target object using the bounding box. For example, the bounding box can provide x, y, w, h, z coordinates, where x, y, z may be 3D coordinate information for the upper left corner of the bounding box in the image (I), w may be the width of the bounding box, and h may be the height of the bounding box. Therefore, it is possible to specify the location of the target object using the coordinate information of the bounding box, and to determine the size of the area corresponding to the target object based on the width and height of the bounding box.

[0065] Additionally, when the image output device (A) outputs a virtual reality image, the image output device (A) can update the image (I) output according to the movement of the user (1). For example, a different image (I) can be output depending on the line of sight of the user (1), and objects displayed in the image (I) can also change depending on the line of sight of the user (1). At this time, the object analysis unit (110) can continuously recognize each target object included in the updated image (I).

[0066] However, depending on the embodiment, object recognition may be performed only when the image output device (A) does not move for a reference time or longer, or only when the object size is greater than a reference size. That is, in order to increase the object recognition speed and reduce unnecessary operations, it is possible to wait until the virtual reality image is maintained to a certain extent, and to omit object recognition for objects smaller than a reference size that are unlikely to interact with the user.

[0067] Thereafter, the object analysis unit (110) can extract the color, size, shape, movement, distance within the image, etc. of the target object from the target object image and generate target characteristic information. The object analysis unit (110) may further include separate machine learning or neural network-based extraction models for extracting characteristic information, and can use these to extract information on the color, size, shape, movement, etc. of the target object from the target object image. For example, information on color, saturation, brightness, etc. distributed within the target object image can be extracted based on pixel information and generated as characteristic information on color. In addition, the direction and amount of movement of pixels between consecutive frames within the image can be obtained, and the movement or vibration magnitude of the target object can be extracted based on this and generated as characteristic information on movement. In addition, it is possible to extract various information related to the sense of touch that can be obtained from visual information on the target object image as object characteristic information.

[0068] For example, the object analysis unit (110) can extract object characteristic information, such as roughness or smoothness, from the surface shape of the object displayed in the object image. In this case, the texture, such as the roughness or smoothness of the surface, can be expressed as a tactile sensation.

[0069] In addition, the color, saturation, brightness, etc. of the target object displayed in the target object image can be extracted as target characteristic information, and in this case, the temperature of the target object can be estimated from the color of the target object, etc., and the temperature can be expressed as a feeling of touch.

[0070] When recognizing the corners or vertices of a target object and extracting the degree of sharpness as target characteristic information, it is possible to express the tactile sensation such as pressure or penetration caused by the target object, and when extracting the movement or vibration of the target object as target characteristic information from consecutive frames, it is possible to reproduce the tactile sensation according to the vibration of the target object.

[0071] Additionally, when extracting information about the distance to the target object as target characteristic information, it is also possible to reproduce the tactile sensation of temperature, wind speed, vibration difference, etc. according to distance.

[0072] Meanwhile, the object analysis unit (110) can generate and set an identifier (ID) for each target object in order to distinguish and track each target object. That is, the same objects may appear repeatedly in the image, and an identifier for each target object can be set in order to track and manage the location of the same object, etc. Depending on the embodiment, the object analysis unit (110) can receive and set an identifier for each target object (OB1, OB2, OB3) from the object tracking unit (120).

[0073] The object tracking unit (120) may include an object tracking database, in which identifiers and location information corresponding to each object may be stored. That is, by utilizing the object tracking database, it is possible to track the location of each object within an image.

[0074] The object analysis unit (110) can provide the object tracking unit (120) with characteristic information of each target object or coordinate information of a bounding box, and the object tracking unit (120) can check whether an object matching the target object transmitted from the object analysis unit (110) is already stored based on each characteristic information or coordinate information.

[0075] Here, the object tracking unit (120) can generate and assign a new identifier when a new target object is input from the object analysis unit (110), and can update the location information while maintaining the same identifier in the case of the same target object. The object tracking unit (120) can determine the identity between each object by utilizing a separate machine learning model or neural network model. Here, the identity between the objects can be determined based on the location and size of each object, or by comparing characteristic information or target object images.

[0076] The object change analysis unit (130) can search for an object corresponding to a target object by comparing the characteristic information of a plurality of objects stored in advance with the target characteristic information, and can extract and provide the target tactile information of the target object from the tactile information of the searched object. Here, the characteristic information of each object can be stored in the object meta storage unit (140), and the tactile information of each object can be stored in a tactile information database (not shown).

[0077] Specifically, the object change analysis unit (130) can obtain similarity by comparing the characteristic information of each object stored in the object meta storage unit (140) with the object characteristic information of the target object. Here, there may be an object whose similarity with the object characteristic information of the target object is greater than a set value, and in this case, the tactile information corresponding to the object whose similarity is greater than the set value can be extracted as the object tactile information of the corresponding target object.

[0078] On the other hand, there may be cases where there is no object in the object meta storage (140) whose similarity with the target object's target characteristic information is greater than or equal to a set value. In this case, the object change analysis unit (130) may request the tactile analysis unit (150) to generate target tactile information corresponding to the corresponding target characteristic information. That is, since the tactile information of objects previously generated cannot be utilized, a request may be made to newly generate tactile information for the corresponding target object. Here, if the tactile analysis model (M) of the tactile analysis unit (150) directly generates tactile information, the amount of computation may be relatively large and the processing speed may take a long time. Therefore, the object change analysis unit (130) may first check the similarity with objects in which tactile information is previously stored, and request the tactile analysis unit (150) to newly generate tactile information only when there is no object whose similarity is greater than or equal to a set value.

[0079] Meanwhile, the tactile analysis unit (150) can input target characteristic information into the tactile analysis model (M) to generate target tactile information corresponding to the target object. Specifically, the tactile analysis model (M) can generate tactile information including various types of tactile sensations such as texture, temperature, pressure, and vibration of the target object using the characteristic information.

[0080] In some embodiments, the tactile analysis model (M) can be trained for each object type of each predefined object, and tactile information can be generated by reflecting characteristic information according to the object type. For example, the object types of each object, such as apple, tree, and fire, can be defined in advance, and texture, temperature, pressure, vibration, etc. according to the object characteristic information for apple, tree, fire, etc. can be individually trained in the tactile analysis model (M) to generate tactile information. In this case, since training is performed for each object, it is possible to generate more accurate tactile information for the objects. In other words, the tactile analysis model (M) can generate different tactile information according to the type of each object even if the characteristic information is the same.

[0081] Referring to FIG. 3, the tactile analysis model (M) may include individual tactile models (M1, M2, M3, M4, M5) corresponding to multiple tactile types. That is, tactile information may be generated using the individual tactile models (M1, M2, M3, M4, M5) corresponding to each tactile type. As illustrated in FIG. 3, the individual tactile models may include a texture analysis model (M1), a heat analysis model (M2), a pressure analysis model (M3), a vibration analysis model (M4), and a pain analysis model (M5). However, the present invention is not limited thereto, and additional individual tactile models corresponding to each tactile type may be further included.

[0082] Specifically, the texture analysis model (M1) can generate the surface texture of the target object based on target characteristic information indicating texture such as surface roughness or smoothness. In addition, depending on the embodiment, the target object image can be further considered to generate tactile information about the surface texture, and at this time, it is also possible to generate tactile information about the texture by determining the roughness of the surface, etc. by applying a particle filter to the target object image.

[0083] The thermal analysis model (M2) can generate a tactile sensation for the temperature of a target object based on target characteristic information such as color, saturation, and brightness of the target object. In other words, the higher the color and brightness, such as red or yellow, the higher the temperature is set, and the lower the color and brightness, the lower the temperature is set, so that tactile sensation information can be generated.

[0084] The pressure analysis model (M3) can generate tactile information indicating the sense of touch, such as pressure, caused by the target object based on object characteristic information indicating the degree of sharpness, such as the corners or vertices of the target object. The vibration analysis model (M4) can generate tactile information about vibration from object characteristic information corresponding to the movement or vibration of the target object, or can generate tactile information according to the vibration intensity of the target object by applying frequency analysis, wavelet transform, etc. from consecutive frames of target object images.

[0085] The pain analysis model (M5) can generate tactile information, representing the sensations of pressure or pain caused by a target object, based on object characteristic information representing the sharp information of the target object. At this time, the pain analysis model (M5) can apply human body modeling and biomechanics technology to predict stress or load on specific body parts, extract the corresponding pain magnitude, and generate corresponding tactile information.

[0086] Thereafter, the tactile information generated from each individual tactile model (M1, M2, M3, M4, M5) can be stored in a tactile information database (not shown).

[0087] Meanwhile, depending on the embodiment, the object change analysis unit (130) may obtain the similarity for each item included in the characteristic information and the target characteristic information. For example, the similarity may be obtained by comparing the characteristic information on shape or size, the characteristic information on color, and the characteristic information on movement. Here, if the similarity is less than a set value, the generation of tactile information for the corresponding item may be requested, and the tactile analysis unit (150) may input the object characteristic information into an individual tactile model (M1, M2, M3, M4, M5) of a tactile type corresponding to the corresponding item, and generate tactile information for each tactile type. For example, among the tactile types, the similarity of texture, pressure, and vibration may be greater than the set value, but the similarity of temperature may be less than the set value. In this case, since there is no need to generate tactile information separately for texture, pressure, and vibration, they may be used as is, and a request may be made to generate tactile information corresponding to the individual tactile model for temperature. That is, the amount of computation can be minimized by performing computations only for the necessary types of touch.

[0088] Additionally, depending on the embodiment, the object change analysis unit (130) may first compare the identifier corresponding to each object with the identifier corresponding to the target object, and if there is an object having an identifier that matches the identifier of the target object, it may compare the characteristic information of the object with the target characteristic information of the target object to obtain the similarity. That is, there may be cases where the same object changes over time or due to a specific event, and there may be cases where the tactile sensation provided to the user changes accordingly. In order to reflect this, the location of objects having the same identifier can be tracked using the object tracker (120), and the characteristic information of the previous object can be compared with the characteristic information of the current object to check whether a difference exists.

[0089] Here, if an object with the same identifier exists and the difference between the object's characteristic information and the target object's characteristic information is no greater than a threshold value, the same tactile information as the object can be provided. Conversely, if the difference between the characteristic information is greater than the threshold value, new tactile information for the object can be generated.

[0090] Additionally, depending on the embodiment, there may be cases where no object with a matching identifier exists. This is the case when a new object is added to the image. In this case, target characteristic information can be input into the tactile analysis model to generate new target tactile information corresponding to the target object.

[0091] In addition, there may be cases where auditory information, olfactory information, taste information, etc. are additionally included in the image in response to the target object. In this case, the object change analysis unit (130) may further utilize at least one of these to extract target tactile information.

[0092] For example, in a video, a user (1) may be playing a baseball game and hitting a ball with a baseball bat. At this time, the impact sound generated when the baseball ball collides with the baseball bat may be output, and the tactile sensation provided to the user (1) may be set differently depending on the size or type of the impact sound. Here, when the user hits the baseball and makes a hit, there may be a hitting sound and corresponding target tactile sensation information.

[0093] Afterwards, when the user (1) hits a home run, a relatively loud and cheerful hitting sound may be generated, and at this time, the object change analysis unit (130) can modify the target tactile information of a normal hit based on the hitting sound and extract the target tactile information corresponding to the home run.

[0094] In addition, in cases where a hitting sound corresponding to a miss is output, it is possible to extract the target tactile information corresponding to a miss by modifying the target tactile information according to the hitting sound.

[0095] The tactile information transmission unit (160) can transmit target tactile information to the tactile reproduction device (B) worn by the user (1) when the user pointer (pointer) displayed in the image is located at a point where interaction with the target object is possible. The user (1) can wear the tactile reproduction device (B) on a part of the body, and as illustrated in FIG. 5, a user pointer (HD) corresponding to the body part being worn can be displayed in the image (I). However, depending on the embodiment, it is also possible to implement the body part on which the user (1) wears the tactile reproduction device (B) and the body part indicated by the user pointer differently.

[0096] The user (1) can move the user pointer, and can also position the user pointer at a point where the user pointer can interact with the target object. Here, when the user pointer is positioned on the surface of the target object, a tactile sensation for the surface can be provided, but there may be cases where it is necessary to provide a tactile sensation corresponding to the target object even when the user pointer is a certain distance away from the surface of the target object. For example, if the target object is something that generates heat, such as a stove, or something that generates wind, such as an electric fan, a tactile sensation for heat or wind can be provided even when the user pointer is a certain distance away. Therefore, in addition to cases where the user pointer is positioned at a point overlapping the target object, even when the user pointer is positioned at a point where the user pointer can interact, the tactile information transmission unit (160) can transmit target tactile information to the tactile reproduction device (B).

[0097] Meanwhile, the tactile information transmission unit (160) can provide target tactile information corresponding to the corresponding interactable point to the tactile reproduction device (B), thereby allowing the user to reproduce the tactile sensation corresponding to the corresponding object. As illustrated in FIG. 5, the user (1) can position the user pointer (HD) on the campfire (OB1), and in this case, the tactile reproduction device (B) can reproduce the tactile sensation corresponding to the temperature of the campfire (OB1) and provide it to the user (1). Depending on the embodiment, the distance between the user pointer (HD) and the campfire (OB1) can be set in the image, and the tactile reproduction device (B) can determine the temperature according to the distance and provide the tactile sensation according to the temperature.

[0098]

[0099] Figure 6 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

[0100] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a tactile information providing device (100) according to embodiments of the present invention. The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may cause the computing device (12) to operate according to the exemplary embodiments mentioned above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, which, when executed by the processor (14), may be configured to cause the computing device (12) to perform operations according to the exemplary embodiments.

[0101] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that is accessible by the computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0102] A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

[0103] The computing device (12) may also include one or more input / output interfaces (22) that provide interfaces for one or more input / output devices (24) and one or more network communication interfaces (26). The input / output interfaces (22) and the network communication interfaces (26) are connected to the communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) via the input / output interfaces (22). Exemplary input / output devices (24) may include input devices such as pointing devices (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and / or photographing devices, and / or output devices such as display devices, printers, speakers and / or network cards. The exemplary input / output devices (24) may be included within the computing device (12) as a component constituting the computing device (12), or may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0104]

[0105] Figure 7 is a flowchart illustrating a method for providing tactile information according to one embodiment of the present invention. Here, each step of Figure 7 may be performed by a tactile information providing device according to one embodiment of the present invention.

[0106] Referring to FIG. 7, the tactile information providing device can recognize a target object included in a video being played (S10). The tactile information providing device can recognize preset target objects using an object recognition model, and the object recognition model may be implemented using machine learning or a neural network. In other words, the object recognition model may be trained to recognize target objects included in an video using learning data corresponding to each target object and generate a bounding box for the corresponding target object.

[0107] Thereafter, the tactile information providing device can generate a target object image by cropping the interior of the bounding box generated corresponding to the location of the target object within the image. In other words, since performing analysis on the entire image may increase unnecessary computational load, the amount of computation required for object analysis can be minimized by generating a target object image corresponding to the desired target object and performing analysis limited to the target object image.

[0108] Here, the tactile information providing device can identify the position and size of the target object using the bounding box. For example, the bounding box can provide x, y, w, h, z coordinates, where x, y, z may be 3D coordinate information for the upper left corner of the bounding box in the image, w may be the width of the bounding box, and h may be the height of the bounding box. Therefore, it is possible to specify the position of the target object using the coordinate information of the bounding box, and to identify the size of the area corresponding to the target object based on the width and height of the bounding box.

[0109] Meanwhile, the tactile information providing device can generate and set identifiers for each target object in order to distinguish and track each target object. That is, the same objects may appear repeatedly in the image, and identifiers for each target object can be set in order to track and manage the locations of the same objects. Depending on the embodiment, an object tracking database may be included, and the object tracking database may store identifiers and location information corresponding to each object. That is, it is possible to track the locations of each object in the image by utilizing the object tracking database. When a new target object is input, a new identifier is generated and assigned, and in the case of the same target object, the location information can be updated while maintaining the same identifier.

[0110] Thereafter, the tactile information providing device can generate object characteristic information for estimating the tactile sensation of the target object from the object image corresponding to the target object (S20). That is, the color, size, shape, movement, distance within the image, etc. of the target object can be extracted from the target object image and generated as object characteristic information. The tactile information providing device can further include separate machine learning or neural network-based extraction models for extracting characteristic information, and can extract information on the color, size, shape, movement, etc. of the target object from the target object image using these. For example, information on color, saturation, brightness, etc. distributed within the target object image can be extracted based on pixel information and generated as characteristic information on color. In addition, the direction and amount of movement of pixels between consecutive frames within the image can be obtained, and the movement or vibration magnitude of the target object can be extracted based on this and generated as characteristic information on movement. In addition, it is possible to extract various information related to the tactile sensation that can be obtained from visual information on the target object image as object characteristic information.

[0111] The tactile information providing device can search for an object corresponding to a target object by comparing the characteristic information of a plurality of objects stored in advance with the target characteristic information, and extract the target tactile information of the target object from the tactile information of the searched object (S30).

[0112] Specifically, the tactile information providing device can compare the characteristic information of each stored object with the target characteristic information of the target object to obtain a similarity. Here, there may be an object whose similarity with the target characteristic information of the target object is greater than a set value, and in this case, the tactile information corresponding to the object whose similarity is greater than the set value can be extracted as the target tactile information of the corresponding target object.

[0113] On the other hand, there may be cases where there is no object whose similarity with the target object's characteristic information exceeds a set value. In this case, the tactile information provider may request the tactile analysis model to generate target tactile information corresponding to the target characteristic information. In other words, since the tactile information of previously generated objects cannot be utilized, a request may be made to generate new tactile information for the target object.

[0114] Specifically, the tactile analysis model may generate tactile information including various tactile types such as texture, temperature, pressure, and vibration of a target object using characteristic information. In some embodiments, the tactile analysis model may be trained for each object type of each predefined object, and tactile information may be generated by reflecting characteristic information according to the object type. For example, the object types of each object, such as apple, tree, and fire, may be defined in advance, and texture, temperature, pressure, vibration, etc. according to the object characteristic information for apple, tree, and fire may be individually trained in the tactile analysis model to generate tactile information. In this case, since training is performed for each object, it is possible to generate more accurate tactile information for the corresponding objects.

[0115] Here, the tactile analysis model may include individual tactile models corresponding to multiple tactile types, and the tactile information providing device may obtain similarity for each item included in the characteristic information and the target characteristic information, and then input the target characteristic information into the individual tactile model of the tactile type corresponding to the item whose similarity is less than a set value, thereby generating tactile information for the tactile type.

[0116] For example, the similarity can be obtained by comparing the characteristic information about shape or size, characteristic information about color, and characteristic information about movement, respectively. Here, if the similarity is less than a set value, the generation of tactile information for the corresponding item can be requested, and the target characteristic information can be input as an individual tactile model of the tactile type corresponding to the corresponding item, so that tactile information for each tactile type can be generated. For example, among the tactile types, the similarity of texture, pressure, and vibration may be greater than the set value, but the similarity of temperature may be less than the set value. In this case, since there is no need to generate tactile information separately for texture, pressure, and vibration, they can be used as is, and the generation of tactile information corresponding to the individual tactile model for temperature can be requested. In other words, since the operation is performed only for the necessary tactile types, the amount of operation can be minimized.

[0117] Additionally, depending on the embodiment, the tactile information providing device may first compare the identifier corresponding to each object with the identifier corresponding to the target object, and if there is an object having an identifier that matches the identifier of the target object, it may compare the characteristic information of the object with the characteristic information of the target object to obtain the similarity. That is, there may be cases where the same object changes over time or due to a specific event, and there may be cases where the tactile sensation provided to the user changes accordingly. In order to reflect this, the locations of objects having the same identifier can be tracked, and the characteristic information of the previous object can be compared with the characteristic information of the current object to determine whether a difference exists.

[0118] Here, if an object with the same identifier exists and the difference between the object's characteristic information and the target object's characteristic information is no greater than a threshold value, the same tactile information as the object can be provided. Conversely, if the difference between the characteristic information is greater than the threshold value, new tactile information for the object can be generated.

[0119] Additionally, depending on the embodiment, there may be cases where no object with a matching identifier exists. This is the case when a new object is added to the image. In this case, target characteristic information can be input into the tactile analysis model to generate new target tactile information corresponding to the target object.

[0120] In addition, there may be cases where auditory information, olfactory information, gustatory information, etc. are additionally included in the image in response to the target object. In this case, the tactile information providing device may further utilize at least one of these to extract target tactile information.

[0121] Thereafter, the tactile information providing device can transmit target tactile information to the tactile reproduction device worn by the user when the user pointer displayed in the image is located at a point where interaction with the target object is possible (S40). The user can wear the tactile reproduction device on a part of the body, and a user pointer corresponding to the body part being worn can be displayed in the image. At this time, the user can move his / her user pointer and position the user pointer at a point where interaction with the target object is possible. Here, when the user pointer is located on the surface of the target object, a tactile sensation for the surface can be provided, but there may be cases where it is necessary to provide a tactile sensation corresponding to the target object even when the user pointer is a certain distance away from the surface of the target object. For example, if the target object is something that generates heat, such as a stove, or something that generates wind, such as an electric fan, a tactile sensation for heat or wind can be provided even when the user pointer is a certain distance away. Accordingly, the tactile information providing device can transmit target tactile information to the tactile reproduction device even when the user pointer is located at an interactable point, other than when it is located at a point overlapping the target object.

[0122]

[0123] The present invention described above can be implemented as computer-readable code on a medium recording a program. The computer-readable medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. Furthermore, the medium may be a variety of recording or storage means, including a single or multiple hardware components, and is not limited to media directly connected to a computer system, but may also be distributed across a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and ROM, RAM, flash memory, and other media configured to store program instructions. Furthermore, other examples of media include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc. Therefore, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

[0124]

[0125] The present invention is not limited to the above-described embodiments and the attached drawings. It will be apparent to those skilled in the art that components of the present invention can be substituted, modified, and altered without departing from the technical spirit of the present invention.

Claims

1. A method for providing tactile information of a tactile information providing device that provides tactile information to reproduce a tactile sensation corresponding to an image being viewed by a user, A step of recognizing a target object contained in a video being played; A step of generating target characteristic information for estimating the tactile sensation of the target object from a target object image corresponding to the target object; and A method for providing tactile information, comprising the step of comparing the characteristic information of a plurality of objects stored in advance with the target characteristic information to search for an object corresponding to the target object, and extracting the target tactile information of the target object from the tactile information of the searched object.

2. In the first paragraph, the step of recognizing the target object is A method for providing tactile information, wherein the location of the target object within the image is tracked by storing identifiers and location information corresponding to the target objects in an object tracking database.

3. In the first paragraph, the target object image is A method for providing tactile information, wherein the tactile information is generated by cropping the interior of a bounding box generated in response to the location of the target object in the image.

4. In the first paragraph, the step of generating the target characteristic information is A method for providing tactile information, wherein at least one of the color, size, shape, movement, and distance within the image of the target object is extracted from the target object image to generate the target characteristic information.

5. In the first paragraph, the step of extracting the target tactile information is A method for providing tactile information, wherein the similarity is obtained by comparing the characteristic information of the plurality of objects stored in advance with the target characteristic information, and the tactile information corresponding to an object whose similarity is greater than a set value is extracted as the target tactile information of the target object.

6. In the fifth paragraph, the step of extracting the target tactile information is A method for providing tactile information, wherein if there is no object with the above similarity greater than the set value, the target characteristic information is input into the tactile analysis model to generate target tactile information corresponding to the target object.

7. In paragraph 6, the tactile analysis model A method for providing tactile information, wherein the tactile information is generated by using the above characteristic information, the tactile information including at least one of the texture, temperature, pressure, and vibration types of the target object.

8. In paragraph 6, the tactile analysis model A method for providing tactile information, wherein each predefined object is learned by object type and the tactile information is generated by reflecting characteristic information according to the object type.

9. In paragraph 8, The above tactile analysis model includes each individual tactile model corresponding to at least one tactile type, The step of extracting the above target tactile information is A method for providing tactile information, wherein the similarity is calculated for each item included in the above characteristic information and the above target characteristic information, and if there is an item whose similarity is less than a set value, the item whose similarity is less than the set value is input into an individual tactile model of a tactile type corresponding to an item whose similarity is less than the set value among each of the individual tactile models, thereby generating tactile information for the tactile type.

10. In the 6th paragraph, the step of extracting the target tactile information is A method for providing tactile information, wherein the method compares each identifier corresponding to the plurality of objects with an identifier corresponding to the target object, and if an object having an identifier matching the identifier of the target object exists among the plurality of objects, the similarity is obtained by comparing the characteristic information of the object with the target characteristic information of the target object.

11. In the 10th paragraph, the step of extracting the target tactile information is A method for providing tactile information, wherein if an object having the above matching identifier does not exist, the target characteristic information is input into a tactile analysis model to generate target tactile information corresponding to the target object.

12. In the first paragraph, the step of extracting the target tactile information is A method for providing tactile information, wherein the target tactile information is extracted by further using at least one of auditory information, olfactory information, and gustatory information included in the image corresponding to the target object.

13. In paragraph 1, A method for providing tactile information, further comprising a step of transmitting the target tactile information to a tactile reproduction device worn by the user when the user pointer displayed in the image is located at a point where interaction with the target object is possible.

14. A computer program stored on a medium for executing a method for providing tactile information according to any one of claims 1 to 13 in combination with hardware.

15. A tactile information providing device including a processor and providing tactile information for reproducing a tactile sensation corresponding to an image being viewed by a user, The above processor Recognizing a target object contained within a video being played; Generating target characteristic information for estimating the tactile sensation of the target object from a target object image corresponding to the target object; and A tactile information providing device that compares the characteristic information of a plurality of objects stored in advance with the target characteristic information to search for an object corresponding to the target object, and extracts the target tactile information of the target object from the tactile information of the searched object.

Images