Catch light encoding device and catch light decoding device

Abstract

People obtain a lot of information from their faces when they communicate. In particular, the eye is considered to provide a large amount of information. Catch light reflected on the pupil of the eye has already been developed as an expression technique for assisting or enhancing the expression of a character in media or the like, and is widely used as an indispensable non-language tool. To simply and appropriately process the shape of the catch light of a character expressed by media or the like by technologically classifying it and encoding or decoding it. As a result, the shape of the catch light can be made into an operable information technology.SOLUTION: The shape of the catch light of the pupil image is encoded and the encoded code is decoded by using a catch light encoding device and a catch light decoding device to generate the pupil image forming the shape of the catch light.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a catchlight shape encoding and decoding device that performs image analysis and calculation processing of the shape of a catchlight reflected in the pupil. [Background technology]

[0002] Previously, there was no way to use information technology to capture the shape of catchlights reflected in the pupils. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Published Patent Application (A) JP2024-45247A [Non-patent literature] [Non-patent literature] "Emotional Expressions through Robot Eye Lighting Patterns Influence Human Social Decision-Making," HAI Symposium 2013, pp. 147-150 Summary of the Invention [Problem to be solved by the invention]

[0004] With the maturation of computer science and rapid advances in AI, humanoid characters, avatars, and humanoids are beginning to play an important role in society as communication media interfaces. It is known that humans obtain a great deal of information from their faces during communication, particularly from their eyes. In traditional media content such as manga and anime, catchlights reflected in the pupils of eyes have developed as an expressive technique that supports and enhances the expression of characters' emotions and worldviews, and are now widely used in everyday life as an indispensable nonverbal tool. Expressions using such catchlight shapes are expected to be widely incorporated into humanoid characters, avatars, and humanoids in digital media, as well as next-generation media interfaces such as mixed reality goggles and mixed reality contact lenses, becoming an important feature in communication. Therefore, this invention aims to enable simple and appropriate processing of the shape of catchlights reflected in the pupils by technologically classifying and encoding the shape elements based on media and psychological research. This aims to make catchlights an interactive and operable information technology that can intervene in communication with human and non-human characters in the information space and space of meaning via media such as monitors, paper, or the eyes of a robot. [Means for solving the problem]

[0005] The basic coding theory of the present invention is based on the results of media engineering and psychological research into catchlights by the inventors, and is characterized by comprising an image input device that takes in a pupil image as an electrical signal and converts it into a processable format, feature extraction processing means that receives image data from the image input device and extracts a feature portion image by comparing it with predetermined pupil data based on feature portion extraction conditions, shape element extraction processing means that extracts shape elements from the feature portion image based on predetermined shape element-characteristic portion association rules to obtain catchlight shape element information, shape element encoding processing means that performs shape element encoding processing on the shape element information based on predetermined shape element code association rules to calculate a shape element code, and a calculation processing device that is equipped with storage means for storing the shape element code, and a shape element code output device that has the calculation processing device and outputs the shape element code.

[0006] The basic decoding theory of the present invention is based on the results of media engineering and psychological research into catchlights by the inventors, and is characterized by comprising a code input device that takes in, as an electrical signal, a catchlight shape element code output by a catchlight shape encoding device and obtains predetermined catchlight shape element code data, shape element decoding processing means that receives the predetermined catchlight shape code data from the code input device and performs shape element decoding processing based on predetermined shape element code association rules to calculate catchlight shape element information, a processing means that obtains predetermined catchlight image data from the obtained shape element information based on association rules between shape elements and characteristic parts, and a calculation processing device that is also equipped with storage means for storing the catchlight image data, and a catchlight image output device that outputs the catchlight image data saved in the calculation processing device. [Effects of the Invention]

[0007] The encoding process of the present invention makes it possible to technologically classify and encode the shapes of catchlights reflected in pupils. This enables simple, accurate, and rapid processing with only necessary, sufficient, and minimal information processing, allowing for rapid and accurate extraction of the desired shape element code. Furthermore, the decoding process of the present invention makes it possible to create and display the desired catchlight shape from the shape element code. Furthermore, if the shape element code is stored in a storage device, the catchlight shape can be accurately restored. This allows for communication technologies and devices that utilize catchlight shapes to be applied, for example, to characters and avatars in media, humanoid eyes, MR goggles, and MR contact lenses. These technologies and devices are expected to not only enrich communication between healthy people, but also help people with speech or hearing impairments, or those with facial muscle immobility, read the emotions of others and express their own. The present invention enables richer and more accurate communication between people, between people and AI, and between AI and AI. [Brief explanation of the drawings]

[0008] Catchlights usually have the same shape reflected in both eyes. Therefore, for ease of explanation, only one eye is shown in the diagram. The up, down, left, and right directions shown in the diagram are those when the target eye is viewed from the outside, front position. [Figure 1] <Number of Shape Elements> This is a diagram of CUs showing shape elements of the characteristic parts of the number of catchlights of the present invention. (a) 1 catchlight (CU 101), (b) 2 catchlights (CU 102), (c) 3 catchlights (CU 103) [Figure 2] This is a diagram showing the shape elements of the characteristic parts of the catchlight position in the eye that has a realistic shape such as a photograph or video, according to the present invention. [Figure 3] This is a diagram showing the shape elements of the characteristic parts of the catchlight position in eyes with shapes created in manga, anime, etc., according to the present invention. [Figure 4] <Shape element of position> This is a diagram of CU showing the shape element of the characteristic part of the catch light position of the present invention. [Figure 5] <Shape Elements of Position> of the Present Invention When a catchlight is displayed across multiple areas, this is a diagram showing shape elements of characteristic parts of the position of the catchlight in the area where the center point of the catchlight is located. [Figure 6] <Shape Elements of Position> This is a diagram showing the shape elements of the characteristic part of the position of the catchlight on the line where the center point of the catchlight is located when the center point of the catchlight is on the boundary line of the area. [Figure 7] <Shape Elements of Position> This is a diagram showing shape elements of the characteristic part of the position of the catchlight at the center point of the catchlight when the center point of the catchlight is on the center point of the area. [Figure 8] <Shape Elements> of the present invention: This is a diagram of CUs showing the shape elements of the characteristic parts of the catchlight shape. (a) Catchlight round (CU 31), (b) Catchlight oval (CU 32), (c) Catchlight square (CU 33), (d) Catchlight hexagonal (CU 34), (e) Catchlight octagonal (CU 35), (f) Catchlight rectangle (CU 36), (g) Catchlight design shape (CU 37). [Figure 9] <Size and Shape Element> This is a diagram of CUs showing the shape elements of the characteristic parts of the size of the catchlight of the present invention. (a) Large catchlight (CU 41), (b) Medium catchlight (CU 42), (c) Medium catchlight (CU 43), (d) Medium catchlight (CU 44), (e) Small catchlight (CU 45). [Figure 10] <Brightness Shape Element> This is a diagram of CUs showing shape elements of characteristic parts of the brightness of catchlights of the present invention. (a) Extremely bright catchlight (CU 51), (b) Bright catchlight (CU 52), (c) Slightly dark catchlight (CU 53), (d) Dark catchlight (CU 54). [Figure 11]<Color Shape Elements> This is a diagram of CUs showing the shape elements of the characteristic parts of the color of catchlights of the present invention. (a) Catchlight Gray (CU 61), (b) Catchlight Pink (CU 62), (c) Catchlight Red (CU 63), (d) Catchlight Purple (CU 64), (e) Catchlight Blue (CU 65), (f) Catchlight Cyan (CU 66), (g) Catchlight Green (CU 67), (h) Catchlight Yellow (CU 68), (i) Catchlight Orange (CU 69). [Figure 12] FIG. 10 is a diagram showing the shape of a catchlight generated by decoding a code encoded using the catchlight encoding device of the present invention. [Figure 13] 1 is a shape element encoding algorithm of the catchlight encoding device of the present invention. [Figure 14] 1 is a flowchart of a catchlight encoding device of the present invention. [Figure 15] 1 is a shape element decoding algorithm of the catchlight decoding device of the present invention. [Figure 16] 1 is a flowchart of the catchlight decoding device of the present invention. 1A is the inverse process of 1, 2A is the inverse process of 2, and 3A is the inverse process of 3. [Figure 17] This is pupil image data input by an image input device. [Figure 18] This is an image of a feature portion extracted using a fixed coordinate system. [Figure 19] This is a pupil image without catchlight for comparative measurement of pupil images. [Figure 20] This is an image of the extracted characteristic area (rectangular screen). [Figure 21] This is the association rules between shape elements, characteristic parts, and shape element codes (part 1). [Figure 22] This is the association rules between shape elements, characteristic parts, and shape element codes (part 2). DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described. 1. Shape element encoding algorithm An image of the pupil is supplied from the image input device 131 to the arithmetic processing device 132 as digital data that can be processed by the arithmetic processing device, and this data is stored and held as pupil image data in the image storage area of ​​the arithmetic processing device 132. Then, according to the flowchart in Fig. 14, the pupil image data (a) is processed in the following order to obtain a shape element code (g). 2. Shape Element Decoding Algorithm The code is supplied from the shape element code input device 151 to the arithmetic processing device 152 as digital data that can be processed by the arithmetic processing device, and this data is stored and held as code data in a code storage area of ​​the arithmetic processing device 152. Then, according to the flowchart of Fig. 16, the shape element code (g) is processed in order as follows to obtain pupil image data (a). 3. Characteristic part extraction processing The feature portion extracted image shown in Figure 18 is obtained by extracting the portions necessary for recognizing catchlights from the pupil image data shown in Figure 17 according to the feature portion extraction conditions, and is compared with Figure 19. By applying the feature portion extraction conditions, extraction is performed using the fixed coordinate method. As a result, the feature portion image (rectangular screen) shown in Figure 20 can be cut out. 4. Shape element extraction processing The changes in the characteristic parts determined based on the shape constituent factors of catchlights identified through media engineering research are associated with the characteristic parts (c) according to the association rules (d) between the shape elements and the characteristic part images, and the shape element information (e) is extracted. Tables for explaining specific examples of the association rules (d) between shape elements and characteristic parts are shown in Figures 21 and 22. 5. Shape element encoding process The code assigned to each shape element is called a catchlight unit. In other words, a shape element code is a unique code assigned to six shape elements: (1) number, (2) position, (3) shape, (4) size, (5) brightness, and (6) color. For example, if there is one catchlight, the code is "CU 101." Furthermore, by performing detailed analysis procedures for the characteristic portion image (c) for each shape element code determined by these shape element code association rules according to the flowcharts in Figures 14 and 16, several measurement values ​​for each shape element code are determined as shape element information (e). The association rules between catchlight shape elements and characteristic portions, and the association rules between shape element codes called catchlight units (CUs) and shape elements are shown below. (1) <Number of Shape Elements> As the rules and means for assessing the change in the number of catchlights between shape elements and characteristic features, the number of catchlights is calculated and assessed by comparing with a required image without catchlights and applying the characteristic feature extraction conditions. Furthermore, an individual identification symbol is assigned to each extracted catchlight as a unique ID. <Number shape element> The CU that evaluates the number of catchlights and the individual identification symbol are shown below. 0 pieces (CU 100), 1 piece (CU 101), 1st piece A, 2 pieces (CU 102), 1st piece A, 2nd piece B 3 pieces (CU 103), 1st piece A, 2nd piece B, 3rd piece C 4 pieces (CU 104), 1st piece A, 2nd piece B, 3rd piece C, 4th piece D 5 pieces (CU 105), 1st piece A, 2nd piece B, 3rd piece C, 4th piece D, 5th piece E 6 pieces (CU 106), 1st piece A, 2nd piece B, 3rd piece C, 4th piece D, 5th piece E, 6th piece F 7 pieces (CU 107), 1st piece A, 2nd piece B, 3rd piece C, 4th piece D, 5th piece E, 6th piece F, 7th piece G 8 pieces (CU 108), 1st piece A, 2nd piece B, 3rd piece C, 4th piece D, 5th piece E, 6th piece F, 7th piece G, 8th piece H 9 pieces (CU 109), 1st piece A, 2nd piece B, 3rd piece C, 4th piece D, 5th piece E, 6th piece F, 7th piece G, 8th piece H, 9th piece J *1 The letter I is not used because it is easily confused with the number 1 and the letter O is not used because it is easily confused with the number 0. *2 If the number of catchlights is 24 or more, the individual identification code will start again with A, with the number of turns at the beginning of the alphabet. (Example) If there are 30 catchlights, it will be 2F. *3 The CU for 100 catchlights is (CU 200). (2) <Shape elements of position> As the rules and means for relating shape elements and characteristic features to evaluate changes in the position of catchlights, catchlights with individual identification symbols are arranged on a two-axis coordinate system with the pupil of the eye as the center point, and the position of the catchlight is evaluated by applying the characteristic feature extraction conditions. <Shape element of position> The CU for evaluating the position of the catchlight is shown below. Top left (CU 21), middle left (CU 22), bottom left (CU 23), top center (CU 24), middle center (CU 25), bottom center (CU 26), top right (CU 27), middle right (CU 28), bottom right (CU 29) *1 If a catchlight is displayed across multiple areas, the position of the catchlight is classified and evaluated in the area where the center point of the catchlight is located (Figure 5, in this case the catchlight position is in the upper left). *2 If the center point of the catchlight is on the boundary line of the area, evaluate the position of the catchlight on the line where the center point of the catchlight is located (Figure 6, in this case the catchlight position is the top center). *3 If the center point of the catchlight is on the center point of the area, evaluate the position of the catchlight at the center point of the catchlight (Figure 7, in this case the position of the catchlight is in the center). (3) <Shape elements of shape> As the rules and means for relating shape elements and characteristic parts to evaluate changes in the shape of catchlights, the shape of catchlights is evaluated by applying characteristic part extraction conditions that determine the approximate shape of catchlights given individual identification symbols. <Shape> The CU for evaluating the shape of the catchlight is shown below. Round (CU 31), oval (CU 32), square (CU 33), hexagonal (CU 34), octagonal (CU 35), rectangular (CU 36), design (CU 37) *1 All shapes other than (CU 31) to (CU 36) are considered design shapes (CU 37). (4)〈Size and shape element〉 The relationship between the shape element that evaluates the change in the size of the catchlight and the characteristic part As a rule and means, the size of a catchlight is evaluated by applying a characteristic portion extraction condition for measuring and determining the size of a catchlight given an individual identification symbol. <Size and shape element> The CU for assessing the size of catchlights is shown below. Large (CU 41): Large enough to cover the entire pupil or almost the entire pupil. Slightly large (CU 42): A size between large and medium. Medium (CU 43): Pupil size is approximately 2mm to 4.5mm. Slightly small (CU 44): A size between medium and small. Small (CU 45): The smallest size that can be seen. (5) <Shape element of brightness> As the rules and means for relating the shape element and characteristic part for evaluating the brightness of a catchlight, the brightness of the catchlight is evaluated by applying the characteristic part extraction conditions for determining the light intensity of brightness for a catchlight given an individual identification symbol. <Brightness element> The CU used to evaluate the brightness of catchlights is shown below. Extremely bright (CU 51): Bright enough to produce a surrounding glow. Bright (CU 52): Completely white. Slightly dark (CU 53): A brightness somewhere between light and dark. Dark (CU 54): The minimum brightness that can be perceived as light. (6) <Color shape element> As the rules and means for relating the shape element and characteristic parts to evaluate the color of a catchlight, the color of the catchlight is evaluated by applying the characteristic part extraction conditions for determining the color of the catchlight using a color chart for the catchlight that has been given an individual identification symbol. <Color shape element> The CU for evaluating the color of catchlights is shown below. Gray (CU 61), Pink (CU 62), Red (CU 63), Purple (CU 64), Blue (CU 65), Cyan (CU 66), Green (CU 67), Yellow (CU 68), Orange (CU 69) [Example]

[0010] 13 shows an embodiment of the "catch-light shape encoding device" of the present invention. This device comprises an image input device 131 that takes in a pupil image as an electrical signal and converts it into a processable format; a processing device 132 that performs image analysis of the pupil image from the image input device, determines catch-light element information, encodes it using a processing device, and stores it; and an output device 133 that outputs the catch-light shape element code calculated by the processing device. The image input device 131 may be a video camera and a computer processing device connected to it; the processing device 132 may be a computer processing device equipped with storage means for storing programs and calculated data; and the shape element code output device 133 may be a numerical / symbol display device, a parallel or serial digital signal output device, or the like. (Specific example of shape element encoding process) There are three catchlights in the eyes, The first one is on the top left, square, large, light, grayish, The second one is below the center, oval, slightly large, slightly dark, grayish, The third one is in the middle right, round, slightly small, dark, greenish, The result of encoding the pupil image of this example using shape elements is as follows: 103, A21, A33, A41, A52, A61, B26, B32, B42, B53, B61, C28, C31, C44, C54, C67 [Example] Figure 15 shows an embodiment of the "catch-light shape decoding device" of the present invention. This device comprises a code input device 151 that receives a catch-light shape element code as an electrical signal and obtains predetermined shape element code data; a processing unit 152 that processes the code input from the code input device to generate and store decoded catch-light pupil image data; and an image output device 153 that outputs the pupil image data stored in the processing unit. The code input device 151 may be a numeric / symbol display device, a parallel or serial digital signal input device, or a video camera and a connected computer processing unit. The processing unit 152 may be a computer processing unit equipped with storage means for storing programs and calculated data. The image output device 153 may be a video output monitor and a connected computer processing unit. (Specific example of shape element decoding process) 102, A24, A31, A42, A52, A61, B26, B32, B42, B53, B61 The result of decoding is as follows: As shown in Figure 12, there are two catchlights in the pupil; the first is located above the center, round, slightly large, bright, and grayish; the second is located below the center, oval, slightly large, slightly dark, and grayish. [Explanation of symbols]

[0011] 1 eye 2 Eyelids 3. Catchlight 4 Location determination area table 5 Catchlight center point 6 Area Boundaries 7 Area center point and area boundary center 8 Area boundary line, left center 9 Area boundary line, center right 10 Area boundary line, top center 11 Area boundary line bottom center 131 Image input device 132 Processing Unit 133 Shape element code output device 151 Shape element code input device 152 Processing Unit 153 Image output device 201 Feature Area Image (Square Screen) 202 Feature Area Image (Square Screen) 203 Feature Area Image (Square Screen)

Claims

1. an image input device that captures a catchlight image reflected in the pupil as an electrical signal and obtains predetermined catchlight image data; a processing unit having feature extraction processing means for receiving predetermined catch light image data from the image input device and extracting a feature portion image based on specific portion extraction conditions, shape element extraction processing means for extracting shape element information from the feature portion image based on predetermined shape element and feature portion association rules to obtain catch light shape element information, shape element encoding processing means for encoding the shape element information based on predetermined shape element codes and shape element association rules to calculate a shape element code, and storage means for storing the shape element code; a shape element code output device that outputs the shape element code stored in the arithmetic processing device; A catchlight shape encoding device comprising:

2. 2. The catchlight shape coding device according to claim 1, wherein the shape elements include six elements: (1) number, (2) position, (3) shape, (4) size, (5) brightness, and (6) color.

3. a shape element code input device that receives the encoded shape element code of the catch light as an electrical signal and obtains shape element code data of a predetermined catch light; a processing unit having shape element decoding processing means for receiving shape element code data of predetermined catch light from the shape element code input device, and performing shape element decoding processing based on association rules between the predetermined shape element code and shape elements to calculate shape element information of the catch light, further having shape element extraction processing means for performing shape element extraction processing from the obtained shape element information based on association rules between shape elements and feature parts to obtain a predetermined feature part image, processing means for performing feature part extraction processing on the feature part image based on predetermined feature extraction conditions to obtain pupil image data reflecting the catch light, and storage means for storing the pupil image data; a catchlight image output device that outputs the image data of the catchlight stored in the arithmetic processing device; A catchlight shape decoding device comprising:

4. An encoding program that encodes the shape of a catchlight reflected in an eye, an input means for inputting image data of the catchlight; a feature extraction processing means for extracting a feature portion image based on a specific portion extraction condition of the image data; a shape element extraction processing means for extracting shape elements from the characteristic portion image based on predetermined association rules between shape elements and characteristic portions, and obtaining shape element information of the catchlight; a shape element encoding processing means for encoding the shape element information based on a predetermined shape element code association rule to calculate a shape element code; an encoding program for causing the program to function as an output unit for outputting the shape element code;

5. A decoding program for decoding the shape of an encoded catchlight shape element code, a code input means for inputting a shape code of the catchlight; a shape element decoding processing means for decoding shape elements based on predetermined shape element code association rules to calculate shape element information of catchlights; a shape element extraction processing means for obtaining a predetermined characteristic portion image from the shape element information based on a rule for associating predetermined shape elements with characteristic portions; a processing means for extracting characteristic portions from the characteristic portion image based on predetermined characteristic extraction conditions to obtain pupil image data reflecting catchlights; a decoding program for causing the device to function as an output unit for outputting the pupil image data;

Images