Information processing device and program

Abstract

To reduce the time and effort for a user when understanding the meaning of a color shown by an object in comparison to a case where only the color shown by the object is displayed.SOLUTION: An image processing server 20 comprises: a meaning information storage part 25 which stores meaning information set by associating a color shown by an object, a range of an RGB value to be recognized as the color and a display content displayed as information indicating the meaning of the color for each object; an image acquisition part 21 which acquires a photographed image from a user terminal 10; an object specification part 22 which specifies the imaged object and specifies the color shown by the specified object by analyzing the photographed image; a display information generation part 23 which specifies the meaning of the color shown by the object specified by the object specification part 22 by referring to the meaning information, and generates a display image for displaying the object in the photographed image in association with the display content showing the meaning of the color shown by the object; and a UI control part 24 which causes the user terminal 10 to display the display image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , , , , ,

[0007] , ,

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] Generally, since the way a colorblind person perceives the color of an object is different from that of a normal person, the colorblind person cannot accurately obtain color information through vision. For this reason, various techniques have been proposed to correctly convey color information to colorblind people.

[0003] For example, in Patent Document 1, a technique has been proposed in which the color name at a specified position in image data is arranged at that specified position in a layer different from the layer displaying the image data.

[0004] Also, in Patent Document 2, a technique has been proposed in which the color for each region in a color image is discriminated, and the discriminated color is synthesized and provided in that region.

Prior Art Documents

Patent Documents

[0005] <00000​​​​​​​​​​​​​​​​​​​​​​​​

[0008] The information processing apparatus according to the present invention comprises a processor, the processor acquires a color image, identifies an object in the acquired color image and the color represented by the object, Based on semantic information that associates multiple combinations of information indicating a range of colors and information indicating the meaning represented by that range of colors for a single object, multiple meanings associated with the identified object are displayed. In the acquired color image, the object at , A region containing colors belonging to the color range associated with the meaning selected by the user from among the multiple meanings displayed, and the user selected It is characterized by displaying meaningful information in an associated manner.

[0009] Furthermore, the processor is characterized by identifying the object by referring to information input by the user.

[0010] Furthermore, the processor is characterized in that, when multiple candidates for the object are extracted from the color image, it presents the candidate objects to the user and identifies the object selected by the user from among the candidate objects as the object in the color image.

[0011] Furthermore, the processor is characterized by using information describing the content of the color image input by the user as information for identifying the object.

[0012] Furthermore, the processor is characterized in that, if the color image is a moving image, it estimates the state of an object in the color image by analyzing the time-series changes of the moving image, and uses the estimated state of the object as information for identifying the object.

[0013] Furthermore, the processor is characterized by controlling the display of information indicating the meaning of the color represented by the identified object, according to the meaning of the color of the object.

[0015] Furthermore, the processor is characterized in that, when the object is food, if a portion of the food is in an unsuitable state, it refers to the semantic information and displays information indicating that the portion is unsuitable for consumption as information indicating the meaning of the color represented by the object, associating it with the portion in the color image.

[0016] Further, when the object is food, if a portion in a state suitable for eating is included in the food, the processor displays, as information indicating the meaning of the color indicated by the object, information indicating that the portion is suitable for eating, in association with the portion in the color image.

[0017] Further, the processor controls the display of information indicating the meaning of the color indicated by the object according to the user's perception of color.

[0018] Further, the processor The color of the object identified from the aforementioned color image corresponds to the color of the aforementioned object controls the display color of display content according to the user's perception of color.

[0019] Further, the processor controls the display color of an object in the color image according to the user's perception of color.

[0020] The program according to the present invention causes a computer to function to acquire a color image, identify an object and the color of the object in the acquired color image, A function that displays multiple meanings associated with a specific object, based on semantic information that is a combination of information indicating a range of colors and information indicating the meaning that the range of colors represents. and for an object in the acquired color image at to A region containing colors belonging to the color range associated with the meaning selected by the user from among the multiple meanings displayed, and the user selected display information indicating the meaning associated therewith.

Advantages of the Invention

[0021] According to the invention described in claim 1, compared with the case where only the color indicated by the object is displayed, it is possible to reduce the effort required for the user to understand the meaning of the color indicated by the object.

[0022] According to the invention described in claim 2, even when an object cannot be identified only by analyzing a color image, it becomes possible to identify the object by referring to information from the user.

[0023] According to the invention described in claim 3, it is possible to cause the user to select the correct object.

[0024] According to the invention described in claim 4, the information input by the user can be effectively utilized to identify an object.

[0025] According to the invention described in claim 5, the information obtained from a moving image can be effectively utilized to identify an object.

[0026] According to the invention described in claim 6, information indicating the meaning of the color shown by an object can be selectively displayed according to the meaning of the color shown by the object.

[0028] Claim 7 According to the invention described therein, a user can be notified of a portion that is not in a state suitable for eating.

[0029] Claim 8 According to the invention described therein, a user can be notified of a portion that is in a state suitable for eating.

[0030] Claim 9 According to the invention described therein, information can be provided in conformity with the way a user viewing a color image sees.

[0031] Claim 10 According to the invention described therein, display content can be displayed in colors that are easy for a user to distinguish.

[0032] Claim 11 According to the invention described therein, inconveniences caused by the user having difficulty distinguishing colors can be eliminated.

[0033] Claim 12 [ According to the invention described therein, compared with the case where only the color shown by an object is displayed, the labor involved for the user in understanding the meaning of the color shown by the object can be reduced.

Brief Description of the Drawings

[0034] [Figure 1]This is a block diagram showing the image processing system in this embodiment. [Figure 2] This figure shows an example of the data structure of semantic information stored in the semantic information storage unit in this embodiment. [Figure 3] This is a flowchart showing the display image generation process in this embodiment. [Figure 4] (a) is a diagram showing an example of a captured image, and (b) is a diagram showing a composite image generated by compositing display content onto the captured image, as it appears to the user. [Figure 5] (a) is a diagram showing an example of a captured image, and (b) is a diagram showing a composite image generated by combining the captured image with other display content, as it is visible to the user. [Figure 6] (a) is a diagram showing an example of a captured image, and (b) is a diagram showing an example of a case where the user is allowed to select an object in the captured image. [Figure 7] (a) is a diagram showing an example of a captured image, and (b) is a diagram showing an example where the user is allowed to select which meaning of the colors of objects in the captured image to display. [Figure 8] This figure shows an example of a display image when the user selects "honey leak" as the meaning of the color represented by the apple in this embodiment. [Figure 9] This figure shows an example of a display image when mold is selected by the user as the meaning of the color represented by the apple in this embodiment. [Modes for carrying out the invention]

[0035] Hereinafter, preferred embodiments of the present invention will be described based on the drawings.

[0036] Figure 1 is a block diagram showing the image processing system in this embodiment. Figure 1 shows an image processing system having a user terminal 10 and an image processing server 20. The user terminal 10 and the image processing server 20 are connected to communicate bidirectionally via various types of networks, such as the Internet, a LAN (Local Area Network), or a network constructed by combining these. The image processing system may include multiple user terminals 10, but each only needs to have equivalent functions as described later, so for convenience, only one user terminal 10 is shown in Figure 1.

[0037] The user terminal 10 is a terminal device used by a user to view images. The user terminal 10 is, for example, a smartphone, a tablet, or a glasses-type device. The user terminal 10 can be implemented using a conventional hardware configuration that includes a processor, storage means, communication means, etc., but in this embodiment, it is necessary to have the function to display color images on the screen as color images. Furthermore, the user terminal 10 in this embodiment has a camera or other means of capturing images and also has the function of generating color images.

[0038] In this embodiment, the user of the user terminal 10 (i.e., the "user") is assumed to be a person with color vision deficiency. "Person with color vision deficiency" refers to anyone other than a person with normal color vision (hereinafter referred to as "person with normal color vision"), and is used as a general term for people with total color blindness, color blindness, and color vision deficiency. Of course, the information provided in this embodiment, namely the information showing the colors of objects, can also be useful information for people with normal color vision, so people with normal color vision may also use the user terminal 10.

[0039] As shown in Figure 1, the user terminal 10 includes a shooting unit 11, an image transmission unit 12, and a UI (user interface) unit 13. The shooting unit 11 generates a color image by taking a picture using a shooting means installed in the user terminal 10. The image transmission unit 12 transmits the color image generated by the shooting unit 11 to the image processing server 20. The UI unit 13 is implemented by a user interface such as an LCD touch panel installed in the user terminal 10, and accepts user input operations and displays images prepared on the user terminal 10 or images transmitted from the image processing server 20.

[0040] Incidentally, as mentioned above, the imaging unit 11 captures an object as the target of imaging in response to user operation and generates a color image. An "object" is generally defined as a substance that has spatial size and shape. In the case of food, objects include, for example, fruits, meat, and fish, and each of these may have many varieties. Also, in the case of fish, for example, even the same species of fish may have different forms such as grilled fish, boiled fish, and sashimi. In this embodiment, if the type or form is different, they may be treated as different objects.

[0041] Each component 11 to 13 in the user terminal 10 is realized through the cooperative operation of the computer forming the user terminal 10 and the program running on the CPU installed in the computer.

[0042] The image processing server 20 corresponds to the information processing device according to the present invention. The image processing server 20 in this embodiment can be realized with a conventional general-purpose hardware configuration. That is, the image processing server 20 is equipped with a CPU, ROM, RAM, storage means such as a hard disk drive (HDD), and communication means. In addition, a user interface including input means such as a mouse and keyboard and display means such as a display may be connected in order to perform semantic information maintenance locally.

[0043] The image processing server 20 includes an image acquisition unit 21, an object identification unit 22, a display information generation unit 23, a UI control unit 24, and a semantic information storage unit 25. Components not used in the description of this embodiment are omitted from the figures. The image acquisition unit 21 acquires a color image transmitted from the user terminal 10. The object identification unit 22 analyzes the color image acquired by the image acquisition unit 21 to identify objects contained in the color image, i.e., objects photographed by the photography unit 11. The object identification unit 22 also identifies the color represented by the identified object. The display information generation unit 23 refers to semantic information to identify the meaning of the color represented by the object identified by the object identification unit 22, and generates a display image as display information for displaying the object in the color image, associating the information indicating the meaning of the color represented by that object. The UI control unit 24 controls the display of the display information generated by the display information generation unit 23 to the user terminal 10. In addition, the UI control unit 24 transmits information to the user terminal 10 and receives information specified by the user during the process of generating the display information.

[0044] Figure 2 shows an example of the data structure of semantic information stored in the semantic information storage unit 25 in this embodiment. In order to apply the functions provided by this embodiment, it is necessary to set the required semantic information in the semantic information storage unit 25 in advance. For each object, semantic information is set to indicate the meaning of the color displayed by that object. Specifically, semantic information is set by associating an object with information about the color displayed by that object, "color (range of RGB values)", and information indicating the meaning of that color, "display content".

[0045] Each object is assigned its name and object ID as identification information. In this embodiment, only objects whose name and substance ID are registered in the semantic information "Object (Object ID)" are treated as objects. Color information is set to the color exhibited by the object and the range of RGB values ​​required to recognize that color. More specifically, color information specifies the range of colors that the object can take using RGB values, which are hue information. For example, if an object is identified as an apple, the portion of the apple image that falls within the range of RGB(102,0,0) to RGB(153,51,0) will be recognized as brown. Note that, as illustrated in Figure 2, multiple hue information may be associated with a single object.

[0046] The display content is content that is displayed on the screen of the user terminal 10 in association with the area of ​​a color within an object when the color of an object included in the color image falls within a range of colors specified by hue information. The display content in this embodiment includes symbols and explanatory text. Symbols are symbols that are superimposed on objects included in the color image. When symbols other than letters are used, users may not understand the meaning of the symbol simply by looking at the symbol on the screen; therefore, the explanatory text is text information that explains the meaning of the symbol. As will be described in detail later, for example, if an object is identified as an apple, and there is an area in the image of the apple that is identified as brown, the symbol will be displayed in association with that area identified as brown, and the explanatory text for the symbol will be displayed stating that there is a leak in that area.

[0047] In the semantic information shown in Figure 2, "apple" is set as an example object. Apples have various colors in both their peel and fruit, but in Figure 2, the colors are shown in the range of RGB values. For example, if the object in the color image is identified as an apple, the color of pixels within the apple image area that fall within the range of RGB values ​​(102,0,0)-(153,51,0) is identified as brown. Also, the color of pixels within the apple image area that fall within the range of RGB values ​​(0,0,0)-(119,119,119) is identified as black. Note that the grayscale values ​​shown in Figure 2 are merely examples, and the RGB values ​​should be set individually from the range of 0 to 255 depending on the object, etc. That is, the range of RGB values ​​that represent black, for example, will differ depending on the object and will not necessarily be the same.

[0048] Furthermore, in the semantic information illustrated in Figure 2, the area of ​​the apple image in the color image that is identified as brown is considered to be an area where honey leakage has occurred. In other words, in the semantic information illustrated in Figure 2, the color brown, which represents an apple, signifies honey leakage. Similarly, the color black, which represents an apple, signifies mold.

[0049] Note that, for the sake of explanation, the semantic information illustrated in Figure 2 only shows examples of settings for brown (representing honey leakage) and black (representing mold) as conditions in which an apple is unsuitable for human consumption. However, it is also possible to set colors that represent conditions in which an apple is suitable for human consumption. Furthermore, since there are many varieties of apples, such as Jonathan and Ohlin, instead of setting "apple" as a single object, it may be possible to set different colors for each type of apple.

[0050] Furthermore, Figure 2 illustrates semantic information using "grilled meat" as an object for ease of explanation. The suitability of meat for consumption varies depending on its state, such as raw or grilled. Also, the meaning of the color represented by the object differs depending on the type of meat, such as chicken or beef. Therefore, in practice, it is desirable to set semantic information as different objects for each type and state of meat.

[0051] Each component 21-24 in the image processing server 20 is realized through the coordinated operation of the computer forming the image processing server 20 and the program running on the CPU installed in the computer. Furthermore, the semantic information storage unit 25 is realized using an HDD installed in the image processing server 20. Alternatively, RAM or an external storage means may be used via a network.

[0052] Furthermore, the program used in this embodiment can be provided not only via communication means, but also stored on a computer-readable recording medium such as a CD-ROM or USB memory. The program provided via communication means or recording medium is installed on the computer, and various processes are realized by the computer's CPU executing the program sequentially.

[0053] People with color vision deficiency may have difficulty recognizing, for example, that vegetables have lost their freshness and turned brown due to spoilage. Also, when grilling meat, they may mistakenly believe that meat is cooked through, even if it is undercooked and not yet ready to eat, because they cannot distinguish the color red. While there may be differences in color vision types and individual variations, people with color vision deficiency may have difficulty distinguishing certain colors from others, which can lead to the inconveniences described above.

[0054] Furthermore, it's sometimes unclear what the color of an object means within that object. For example, even if an object is identified as red, the user may have to decide for themselves what the color red means within that object.

[0055] Therefore, in this embodiment, we have made it possible to provide information about what meaning the color an object displays has. Specifically, instead of simply providing the color an object displays, such as "red" or "green," we have made it possible to provide supplementary information about what red or green means in relation to that object.

[0056] Next, the operation in this embodiment will be described. First, the display image obtained as a result of processing by the image processing server 20 in this embodiment will be described.

[0057] Figure 4 shows a color image with grilled meat as an example object. The left side of Figure 4 (a) shows an example of a captured image taken by the user terminal 10. In this embodiment, the captured image is always a color image. The right side of Figure 4 (b) is a display image generated by compositing display content onto the captured image shown in Figure 4(a). The display image is inherently a color image because it is generated by compositing display content onto the captured image. However, Figure 4(b) is shown as an image that can be seen by a person with color vision deficiency (hereinafter referred to as the "recognition image"). The person with color vision deficiency exemplified here is assumed to be a user who has difficulty distinguishing red from other colors.

[0058] Next, in this embodiment, the display image generation process for generating the display image to be shown on the user terminal 10 will be explained using the flowchart shown in Figure 3. Unless otherwise specified, the explanation will use a scene in which a user with color blindness grills and eats meat as an example. Note that the captured image may include other materials, cooking utensils, and other objects, but for the sake of explanation, the objects that will be used to synthesize the display content will be described as being identified from the position of objects included in the captured image or from user specifications. Also, there may be multiple objects rather than one, as illustrated in Figure 4, but for the sake of convenience, they will be described as a single "object" regardless of their number.

[0059] First, the user opens a designated application on the user terminal 10 and uses the camera function of that application to take a picture of the meat being grilled. When the shooting unit 11 on the user terminal 10 generates a picture of the meat in response to the user's operation, the image transmission unit 12 sends the picture to the image processing server 20. In this example, the image is sent to the image processing server 20 using an application on the user terminal 10, but the process of sending the image itself is not unique to this embodiment, and the image can be sent to the image processing server 20 using various existing technologies.

[0060] When a captured image is transmitted from the user terminal 10, the image acquisition unit 21 in the image processing server 20 acquires the captured image (step 110). Subsequently, the object identification unit 22 analyzes the captured image acquired by the image acquisition unit 21 to identify the objects contained in the captured image (step 120). For example, a method may be used in which a neural network such as a CNN (Convolutional Neural Network) is constructed and classification is performed using a deeply trained classifier. As a result of this processing, the object (more precisely, "information that identifies the object" such as its name) and the confidence level of the identified object are output. The "confidence level" here refers to a statistical measure that indicates how certain the object output as a result of the processing is.

[0061] Incidentally, the objects that the neural network outputs as processing results, in other words, the objects that the object identification unit 22 can identify, are the objects that are set and registered in the semantic information shown in Figure 2. That is, the object identification unit 22 can identify objects included in the captured image only if the objects are set in the semantic information. In other words, objects that are set and registered in the semantic information are identified as objects by the object identification unit 22. For this reason, what kind of objects are set and registered in the semantic information is an important factor. As mentioned above, even if the object is a "fish," it is necessary to register the type of fish, such as tuna or flounder, in the semantic information, or even the same type of fish in different forms, such as grilled fish, boiled fish, or sashimi, or to register a combination of these.

[0062] The object identification unit 22 identifies objects included in the captured image as described above, but in some cases it may output multiple candidate objects. If the confidence level of the object with the highest confidence level among the multiple candidate objects is equal to or greater than a predetermined threshold (hereinafter referred to as the "first threshold") (Y in step 130), the object identification unit 22 proceeds to step 160, assuming that it has been able to identify the object from the captured image and uniquely identify it. On the other hand, if the confidence level of the object with the highest confidence level is less than the first threshold (N in step 130), the object identification unit 22 processes the image as follows, assuming that it may not have been able to accurately identify the object from the captured image, in other words, that the object identification accuracy was low.

[0063] Figure 6 shows an example of a captured image generated by a user taking a photograph. This image is based on the assumption that the user took a photograph of fruit. Alternatively, the image processing server 20 may acquire a captured image generated by someone else taking a photograph of the user shopping, via the user terminal 10. When the object identification unit 22 acquires the captured image shown in Figure 6(a), it infers that the user is targeting the fruit enclosed in frame 31 because it is close to the center of the captured image. The object identification unit 22 then identifies the object as described above (step 120), but if the confidence level of the object with the highest confidence level is less than the first threshold (N in step 130), the object identification unit 22 creates a list that includes the name of the object recognized and the confidence level of that object.

[0064] This list contains candidate objects in descending order of confidence. Then, in response to instructions from the object identification unit 22, the UI control unit 24 sends the image with the frame 31 and the list of candidate objects within the frame 31 to the user terminal 10, and displays them on the user terminal 10, thereby presenting the candidate objects to the user (step 140).

[0065] Figure 6(b) shows an example of a screen displayed on the user terminal 10. The user can recognize which object is the target of processing by the frame 31, and can also know from the list of objects 32 that the image processing server 20 has not been able to identify the object. Here, the user selects a radio button 33 from the list 32 that corresponds to the name of the object to be selected. When the UI control unit 24 receives the object corresponding to the radio button 33 selected by the user, the object identification unit 22 can identify what the object with the frame 31 is, that is, the object to be processed among the objects in the captured image, based on the information received by the UI control unit 24 (step 150).

[0066] Figure 6(b) shows an example where three object candidates are presented to the user. However, this is just one example, and it is sufficient for a predetermined number of multiple object candidates to be included in List 32. Alternatively, objects with a confidence level between a threshold value smaller than the first threshold (referred to as the "second threshold") and less than the first threshold may be extracted from the object candidates and included in List 32 for presentation to the user. Furthermore, as illustrated in Figure 6(b), a button 34 labeled "See other results" may be included in the list. When the user selects button 34, the object identification unit 22 may further include object candidates that were not initially included in List 32 and present them to the user. The number of object candidates newly included in List 32 may be a predetermined number as described above, or it may be objects with a confidence level of the second threshold or higher.

[0067] Furthermore, the image processing server 20 assigns and displays the frame 31 to an object inferred from the captured image, but it may also display the frame 31 in a way that allows it to be moved or resized within the captured image. When the user performs an operation to change the frame 31 from its current state (i.e., position and size), the object identification unit 22 changes the object to be processed within the captured image according to the position and size of the frame 31 specified by the user, and identifies the object as described above (steps 120 to 150).

[0068] Furthermore, in order to perform the processing as described above, it is necessary to inform the user in advance that the frame 31 is an area for identifying the object to be processed, and that the frame 31 be displayed in a color that is easily visible to the user.

[0069] The process described above, in which the object identification unit 22 identifies objects in the captured image through its own analysis, or identifies them by querying the user, is merely an example and is not limited to this. For example, a list of objects 32 could be constantly displayed on the user terminal 10, allowing the user to select an object. Alternatively, the user could be asked to directly input information that can identify an object, such as "grilled meat," or to input information that describes the content of the captured image, such as "meat is currently being grilled," as information for identifying the object. In this way, the object could be identified by constantly referring to or supplementarily referring to information input by the user.

[0070] The object identification unit 22 identifies an object in the captured image, and then analyzes the captured image to identify the color of the identified object (step 160). Specifically, within the image region of the object obtained by analyzing the captured image, it finds a part (which may also be expressed as a "range") that corresponds to the color set in the semantic information. If a corresponding part exists, that part is identified as having the color set in the semantic information. As explained here, "the color of the object" is not limited to the color of the entire object, but can also be interpreted as the color of a part of the object. In the example of a captured image of grilled meat shown in Figure 4, multiple pieces of meat are captured, but the multiple pieces of meat can be treated as one object, and each piece of meat can be treated as a part of the object. Alternatively, one piece of meat can be treated as one object, and the color of each part of that piece of meat can be identified. Note that the color of the object identified by the object identification unit 22 must be set in the semantic information, just as when identifying the object itself.

[0071] Next, the display information generation unit 23 obtains display content corresponding to the color shown by the identified object by referring to semantic information (step 170). Then, the display information generation unit 23 generates a display image to be presented to the user by combining the display content with the captured image obtained via the object identification unit 22 (step 180).

[0072] The UI control unit 24 transmits the generated display image to the user terminal 10 in response to instructions from the display information generation unit 23, and displays it on the user terminal 10. Figure 4(b) shows the display image generated in this way as seen by the user.

[0073] For people with normal color vision, it is easy to distinguish between the uncooked parts of meat (which appear red to people with normal color vision) and the cooked parts of meat (which do not appear red to people with normal color vision). However, users who have difficulty distinguishing red will have difficulty visually determining the doneness of meat.

[0074] Therefore, in this embodiment, for people with normal color vision, the displayed content is associated with the red color of the grilled meat in the color image and displayed as information that represents the red color of the grilled meat. Specifically, as illustrated in Figure 4(b), a symbol "×" 41 extracted from semantic information is displayed on the part that appears red, and text information 42 explaining the meaning of the symbol "×" 41 is linked to the displayed image and displayed.

[0075] This allows the user to know that the areas marked with the yakiniku symbol "×" 41 are undercooked. Simply presenting the information that it is "red" would require the user to figure out what that red color means on their own. Therefore, in this embodiment, instead of presenting information that identifies the color "red," information that the color red represents is presented to the user.

[0076] In Figure 4(b), information indicating the meaning of the color displayed by the object is shown by displaying symbol 41 as a "×" shape corresponding to the red area of ​​the object. However, it may also be displayed as text. An example of this display is shown in Figure 5(b). Note that Figures 4(a) and 5(a) are the same image. In the example shown in Figure 5(b), information indicating the meaning of the color displayed by the object is presented to the user by overlaying text on the area of ​​that color. Using text allows the user to intuitively understand the meaning of the color displayed by the object. However, this method is not suitable when short text, such as a single character, is insufficient to convey the meaning of the color, or when overlapping text makes it difficult to read.

[0077] Basically, as illustrated in Figure 2, it is preferable to set semantic information for each object that relates to colors that may be inconvenient for the user, and to present information that will draw the user's attention. Specifically, as illustrated in Figure 2, if the object is food, and there is a part of the food that is unsuitable for eating, semantic information is set to indicate that the part is unsuitable for eating as semantic information indicating the meaning of the color of the food. The image processing server 20 then refers to the semantic information and displays the information indicating that the part is unsuitable for eating as semantic information indicating the meaning of the color of the food, associating it with the part in the captured image.

[0078] Of course, conversely, if a portion of the food is suitable for eating, the semantic information may be set to indicate that the portion is suitable for eating as information indicating the meaning of the color of the food. In this case, the image processing server 20 will refer to the semantic information and display the information indicating that the portion is suitable for eating as information indicating the meaning of the color of the food, associating it with the portion in the captured image.

[0079] Alternatively, both types of information may be set and registered as semantic information, and both types of information may be displayed. Alternatively, the user may be allowed to select which information to display. For example, the image processing server 20 may add a toggle button to the captured image and display it on the user terminal 10, allowing the user to operate the toggle button to select which information to display. In this case, the semantic information must be set in association with each color to determine the display mode selected by the toggle button, i.e., whether or not the object is in a state suitable for eating. Also, although this example uses two display modes, whether or not the object is suitable for eating, if three or more display modes are set, it is necessary to set in association with each color whether or not the object should be displayed when each display mode is applied. In this way, the image processing server 20 may control the display of information indicating the meaning of the color shown by an object according to the meaning of the color shown by the object identified from the captured image.

[0080] However, if the information displayed on the user terminal 10 along with the captured image becomes excessive, the user may have difficulty determining the meaning of the colors represented by the object. Therefore, it may be possible to allow the user to select which information to display. A specific example of this case is explained using Figure 7.

[0081] Figure 7 shows an example of a photograph in which an apple is photographed as an object. When the photograph shown in Figure 7(a) is displayed on the user terminal 10, and the user performs a predetermined operation, the display information generation unit 23 identifies the display content set for the identified object (apple in the example shown in Figure 7) by referring to semantic information, and generates a list that includes information indicating the meaning of the colors represented by the apple that can be presented by that display content.

[0082] Figure 7(b) shows an example of a screen displayed on the user terminal 10 in response to a predetermined operation by the user. The user can recognize the meaning of the colors represented by the apples that can be displayed by referring to the list 35 of information indicating the meaning of the colors represented by the apples that can be displayed. The user then selects from the list 35 the information they want to display in addition to the captured image. Since the user can select one or more pieces of information, they select the checkbox 36 corresponding to the meaning of the color represented by the apple they want to display and then select the send button 37.

[0083] When the UI control unit 24 receives the meaning of the color represented by the apple corresponding to the checkbox 36 selected by the user, the display information generation unit 23 performs image analysis to determine whether the color corresponding to the meaning received by the UI control unit 24 is included in the object in the captured image. If the color corresponding to the meaning received by the UI control unit 24 is included in the object in the captured image, the display information generation unit 23 obtains the display content corresponding to the color included in the object in the captured image by referring to the semantic information, and generates a display image to present to the user by compositing the display content with the captured image as described above. This allows the user to display only the information indicating the meaning of the desired color.

[0084] Figure 8 shows an example of the display image when "seal leak" is selected by the user in Figure 7(b). Figure 9 shows an example of the display image when "mold" is selected by the user, assuming the object is an apple.

[0085] As mentioned above, in this embodiment, information indicating the meaning of the color displayed by an object is displayed using symbol 41. Users can understand the meaning of the color displayed by an object through symbol 41 and the explanatory text (or the meaning represented by the text if symbol 41 is text). However, depending on the display color, it may be difficult for users to distinguish symbol 41. Therefore, it is preferable to display the symbol in a color that is easy for users to distinguish.

[0086] Incidentally, according to the Color Universal Design Organization (CUDO), a specified non-profit organization, there are C-type, P-type, D-type, and A-type human color perception. For example, the P-type, included in Figure 7(b), is a type in which green and red appear to have similar hues. Therefore, it is preferable not to use red and green, which are difficult for P-type users to distinguish, as display colors for the display content. Thus, when the display information generation unit 23 synthesizes the display content with the captured image in step 180 in Figure 3, it is preferable to determine the display colors of the display content according to the user's characteristics, such as C-type or P-type, i.e., how the user perceives color. To this end, the image processing server 20 may, at any stage prior to step 180 in Figure 3, for example, when acquiring the captured image from the user terminal 10 in step 110, have the user specify the user's characteristics and process the acquisition of the user's characteristics along with the captured image. The image processing server 20 then controls the display of information indicating the meaning of the colors shown by objects according to the user's characteristics, i.e., how the user perceives color. Specifically, the display information generation unit 23 determines the display color of the display content to a color that is easily distinguishable by the user, according to the user's declared color perception (e.g., type C).

[0087] By the way, in captured images, if objects with colors that are difficult for the user to distinguish are placed next to each other, the boundary between those objects becomes difficult to discern. Therefore, although it may not be information that indicates the meaning of the colors that the objects represent, the display color of objects in the captured image may be controlled according to the user's perception of colors in order to make the boundaries between objects easier to understand. Specifically, the color of one of the adjacent objects may be changed to a color that is easier for the user to distinguish. Alternatively, the color of an object may be made to a high-contrast color (so-called high-contrast) to make it easier to distinguish the object.

[0088] In this embodiment, as described above, information indicating the meaning of the color shown by an object, i.e., display content, is synthesized and displayed on the captured image. The display content to be synthesized may be such that the user is allowed to select the meaning of the color shown by the object to be displayed, as explained using Figure 7, for example. Focusing on symbol 41, various modifications can be considered for the way symbol 41 is displayed.

[0089] For example, the object identification unit 22 identifies the color of an object based on the RGB values ​​of the color the object displays in the captured image. However, the way the symbol 41 is displayed, that is, the display form of the symbol 41, may be determined based on which level within the RGB value range the color of the object is at.

[0090] For example, according to the semantic information illustrated in Figure 2, even when the color of an apple is specified as black, a range from RGB(0,0,0) to RGB(119,119,119) is provided. Therefore, there is a range from what is called pure black near RGB(0,0,0) to a degree of black that can be considered black for an apple near RGB(119,119,119), and the display form of symbol 41 is set according to the degree of that color. For example, the size, saturation, or brightness of symbol 41 may be changed according to the degree of hue shown by the object, i.e., the RGB value, or the number of symbols 41 displayed in association with the object may be increased or decreased.

[0091] Furthermore, information indicating the meaning of the colors displayed by an object may be appropriately integrated. For example, in the case of an apple, instead of showing the apple's condition in detail, such as whether it is rotten or moldy, the information may be integrated into the fact that it is inedible and displayed as a general term. This can be achieved by assigning the same display content to information about the color of an apple that is inedible (rotten, moldy, etc.).

[0092] By the way, the above explanation assumed that the image captured by the user terminal 10 was a still image, but it may also be a moving image. In the case of a moving image, the state of the object may change. Therefore, the object identification unit 22 may infer the state of the object in the color image by analyzing the time-series changes of the moving image, and use the inferred state of the object as information for identifying the object. For example, if it is meat, the color will change as it is cooked, so from this state it will be easier to identify the object as "grilled meat". In the case of a moving image, the display form of the symbol 41 may be changed according to the change in color shown by the object. For example, the symbol 41 may be gradually made brighter or smaller. For example, if the object in the moving image is "grilled meat", the display of the symbol 41 indicating undercooked meat may be controlled to become smaller as the meat cooks, and finally disappear.

[0093] Furthermore, while the above explanation assumes that the color images processed by the image processing server 20 are images captured by the user terminal 10, it is not necessary to limit this to images captured by the user terminal 10. Also, although the image processing server 20 sends the generated display image to the source of the color image, the source of the color image and the destination of the display image do not necessarily have to be the same. For example, the system may be configured so that an external device such as the user terminal 10 specifies the source of the color image and the destination of the display image.

[0094] In the above embodiment, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0095] Furthermore, the operation of the processor in the above embodiments may not be performed by a single processor, but may be performed by multiple processors located in physically separate locations working together. Also, the order of the processor's operations is not limited to the order described in each of the above embodiments, but may be changed as appropriate. [Explanation of symbols]

[0096] 10 User terminal, 11 Shooting unit, 12 Image transmission unit, 13 UI (User Interface) unit, 20 Image processing server, 21 Image acquisition unit, 22 Object identification unit, 23 Display information generation unit, 24 UI control unit, 25 Semantic information storage unit.

Claims

1. Equipped with a processor, The aforementioned processor, Acquire a color image, Identify the objects and the colors they represent in the acquired color image. Based on semantic information that associates multiple combinations of information indicating a range of colors and information indicating the meaning represented by that range of colors for a single object, multiple meanings associated with the identified object are displayed. In the acquired color image, the system displays the region containing a color that belongs to the color range associated with the meaning selected by the user from among multiple displayed meanings, and associates this with information indicating the meaning selected by the user. An information processing device characterized by the following:

2. The information processing apparatus according to claim 1, characterized in that the processor identifies the object by referring to information input by the user.

3. The aforementioned processor, If multiple candidates for the object are extracted from the color image, the candidates for the object are presented to the user. The object selected by the user from among the candidate objects is identified as the object in the color image. The information processing apparatus according to feature 2.

4. The information processing apparatus according to claim 2, characterized in that the processor uses information describing the content of the color image input by the user as information for identifying the object.

5. The aforementioned processor, If the aforementioned color image is a moving image, the state of the objects in the color image is inferred by analyzing the time-series changes of the moving image. The estimated state of the object is used as information to identify the object. The information processing apparatus according to feature 1.

6. The information processing apparatus according to claim 1, characterized in that the processor controls the display of information indicating the meaning of the color represented by the object, according to the meaning of the color of the identified object.

7. The information processing apparatus according to claim 1, wherein, in the case where the object is food, if a portion of the food is in an unsuitable state, the processor refers to the semantic information and displays information indicating that the portion is unsuitable for consumption as information indicating the meaning of the color represented by the object, in association with the portion in the color image.

8. The information processing apparatus according to claim 1, wherein, when the object is food, if a portion of the food is in a state suitable for eating, the processor refers to the semantic information and displays information indicating that the portion is suitable for eating as information indicating the meaning of the color represented by the object, in association with the portion in the color image.

9. The information processing apparatus according to claim 1, characterized in that the processor controls the display of information indicating the meaning of the color represented by the object according to how the user perceives colors.

10. The information processing apparatus according to claim 1, characterized in that the processor controls the display color of the display content corresponding to the color of the object identified from the color image, according to how the user perceives colors.

11. The information processing apparatus according to claim 1, characterized in that the processor controls the display color of objects in the color image according to how the user perceives colors.

12. On the computer, A function to acquire color images. A function to identify objects and their colors within acquired color images. A function that displays multiple meanings associated with a specific object, based on semantic information that associates multiple combinations of information indicating a range of colors and information indicating the meaning represented by that range of colors for a given object. A function that displays, in relation to an object in an acquired color image, regions containing colors that belong to the color range associated with the meaning selected by the user from among multiple displayed meanings, and information indicating the meaning selected by the user. A program to achieve this.

Images