Image processing system, image processing method, program, and recording medium

Abstract

This invention provides an image processing system, image processing method, program, and recording medium that can display images taken around the time of capture in a timeline, starting from one image in a list of images searched using conditions other than the time of capture. [Solution] In this invention, from a group of images, images with tag information matching the tag information entered by the user are searched for as search images, and from the search images displayed on the display unit, one search image selected by the user is designated as the specified search image. Tag information is information for searching for images from a group of images using tag information other than the shooting time. When a specified search image is specified, only the specified search image is displayed on the display unit, and then, each time a user inputs an instruction, only one image taken before or after the shooting time of the specified search image is sequentially displayed on the display unit from the group of images in chronological order from the shooting time of the specified search image.

Description

Technical Field

[0001] The present invention relates to an image processing system, an image processing method, a program, and a recording medium that search for an image from an image group based on search conditions and display the image as a search result on a display unit.

Background Art

[0002] For example, when searching for a desired image from an image group held in a smartphone, a user uses a method of causing a display unit to display a list (thumbnail images) of images included in the image group in time series and searching for the desired image while tracing back the list of images in time series. Further, a method of inputting a keyword for searching for an image from an image group and searching for an image that matches the keyword using additional information of the image such as tag information is known.

[0003] However, with the method of displaying and tracing back a list of images, it is very difficult to find a desired image from a large number of images included in the image group. Further, with the method of searching for an image by inputting a keyword, if it is clear which keyword to use, it may be possible to appropriately search for the desired image, but if the keyword is not clear, it is very difficult to appropriately search for the desired image.

[0004] Here, Patent Documents 1 and 2 are prior art documents related to the present invention.

[0005] Patent Document 1 describes performing a search for image data by performing a character string search based on additional information attached to the image data. For example, it is described that by performing a search of either AND / OR of a desired Exif character string and a desired note character string, image data having additional information that matches the input character string is searched, and the image data corresponding to the search result is displayed.

[0006] Patent Document 2 describes displaying multiple attributes and attribute values ​​related to a digital image in a label, and allowing the user to select another set of images by selecting one of the attribute values ​​displayed in the label. It also describes that, while viewing the selected set of images, the user can search for all images of a specific person, display those images chronologically, and display their associated labels. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2009-124270 [Patent Document 2] Patent No. 5216591 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Patent Document 1 describes performing an AND search on multiple strings of additional information, but this requires the user to input the string they want to search for, making it impossible to search for images intuitively. Furthermore, while Patent Document 2 describes how to view a set of other images by selecting one attribute value of a single digital image, it only allows viewing a set of images corresponding to a single attribute value, and it is not possible to narrow down images using, for example, multiple attribute values.

[0009] Furthermore, some traditional online photo services have a feature that sends notifications to users encouraging them to view images, prompting them to view past images they own.

[0010] After a notification prompting the user to view images is sent, and the user opens this notification, a collage image created using several images selected from, for example, a group of images owned by the user dated "2015 / 01 / 10" will be displayed on the display unit, as shown in Figure 20.

[0011] Next, when the user presses the "Show more photos" button displayed below the collage image, a group of images taken on the same day as the images used in the collage image, "January 10, 2015 (Saturday)," will be displayed in descending order of the time the images were taken, as shown in Figure 21.

[0012] Furthermore, by moving the displayed image group upwards or downwards, the user can sequentially display images taken on the preceding and succeeding shooting days in chronological order of the images' shooting times. For example, when the group of images displayed on the display moves upward, the group of images taken on a date prior to "January 10, 2015 (Saturday)," such as "January 9, 2015 (Friday)," will be displayed. Conversely, when the group of images displayed on the display moves downward, the group of images taken on a date later than "January 10, 2015 (Saturday)," such as "January 18, 2015 (Sunday)," will be displayed.

[0013] Thus, traditional online photo services have a function that allows users to view a collage image and then click the "Show more photos" button to display a group of images taken on the same day as the images used in the collage image, in chronological order.

[0014] The collage image included in the notification prompting the user to view the image was created using a search criterion related to the image's shooting date, "January 10, 2015 (Saturday)," and selected from a group of images taken on "January 10, 2015 (Saturday)." Therefore, it is possible to display a chronological list of images taken around the time the images used in the collage were taken, all taken on "January 10, 2015 (Saturday)." However, conventional online photo services do not have a function that displays images taken around the time of a particular image in chronological order, based on a search using arbitrary criteria other than the time the image was taken.

[0015] Furthermore, in conventional online photo services, when searching for images matching a keyword from a group of images, the images matching that keyword are displayed in descending order of the time the images were taken, organized by the date they were taken. If you search for images matching the keyword "soccer" within the set of images, for example, as shown in Figure 22, nine images matching soccer will be found within the set of images dated "December 5, 2015 (Saturday)".

[0016] If the "downward arrow" button displayed to the right of "December 5, 2015 (Sat)" is pressed, all images included in the "December 5, 2015 (Sat)" image group, including images unrelated to soccer, will be displayed, and the "downward arrow" button will change to an "upward arrow" button. If the "upward arrow" button is pressed, the nine soccer-related images will be displayed again, and the button will revert to a "downward arrow" button.

[0017] As shown in Figure 22, when searching with the keyword "soccer," the earliest date on which images matching "soccer" were found, prior to December 5, 2015 (Saturday), was November 28, 2015.

[0018] Now, let's return to the time-series image list screen where all images in the image group are displayed in chronological order of when they were taken. If we look at the image group for "December 5, 2015 (Sat)", let's assume that in the time-series image list, the date taken for the image following "December 5, 2015 (Sat)" is "December 6, 2015 (Sun)", and the date taken for the image before "December 5, 2015 (Sat)" is "December 4, 2015 (Fri)".

[0019] Thus, conventional online photo services have a function that, when searching for images using the keyword "soccer" within a group of images, displays images from the same shooting date that contain images related to "soccer," organized by the shooting date.

[0020] However, conventional online photo services do not have a function that, with a single action such as pressing a button, displays in time series the images taken before and after the shooting time of one of the images in the list of images of the hit shooting date. In other words, the user has to remember the shooting date that he or she wants to display in the list of time series images, display the list of time series images, and move the user to the shooting date that he or she wants to display.

[0021] An object of the present invention is to provide an image processing system, an image processing method, a program, and a recording medium that can display in time series the images taken before and after the shooting time of one of the images in the list of images retrieved under conditions other than the shooting time of the images.

Means for Solving the Problems

[0022] In order to achieve the above object, the present invention includes an instruction acquisition unit that acquires an instruction input from a user, an image search unit that searches for an image that matches a search condition input according to an instruction from a user from among an image group as a search image, an image display unit that causes the display unit to display the images included in the image group, and an image designation unit that designates one image selected according to an instruction from a user from among the images displayed on the display unit, and the search condition is a condition for searching for an image from among the image group under conditions other than the shooting time, the image designation unit designates one search image selected according to an instruction from a user from among the search images displayed on the display unit as a designated search image, and when the designated search image is designated, the image display unit causes the display unit to display only the designated search image, and then causes the display unit to display in time series, as time series images, the designated search image and at least a part of the images included in the image group that were taken before and after the shooting time of the designated search image, according to an instruction from a user, and provides an image processing apparatus.

[0023] Here, after the image display unit displays only the specified search image on the display unit, every time an instruction from the user is input, only one search image captured before or after the shooting time of the specified search image is sequentially displayed on the display unit in the order of the search images captured in time series from the shooting time of the specified search image, which is preferable.

[0024] Also, the image specifying unit designates one time-series image selected according to an instruction from the user as a specified time-series image from among the time-series images displayed on the display unit. When the specified time-series image is designated, the image display unit displays only the specified time-series image on the display unit, and then every time an instruction from the user is input, only one image captured before or after the shooting time of the specified time-series image is sequentially displayed on the display unit in the order of the images captured in time series from the shooting time of the specified time-series image, which is preferable.

[0025] Also, the image search unit includes a tag information display unit that displays tag information attached to the images included in the image group on the display unit, a tag information specifying unit that designates tag information selected according to an instruction from the user as selected tag information from among the tag information displayed on the display unit, and an image extraction unit that extracts an image with the selected tag information as a search image from among the image group. The tag information display unit displays at least a part of all the tag information attached to all the images included in the image group on the display unit. The tag information specifying unit designates the first tag information selected according to an instruction from the user as the first selected tag information from among the tag information displayed on the display unit. The image extraction unit extracts an image with the first selected tag information as the first search image from among the image group. The image display unit displays at least a part of all the first search images on the display unit. The tag information display unit preferably displays at least a part of all the tag information attached to all the first search images on the display unit.

[0026] Furthermore, the tag information specification unit simultaneously specifies, in addition to the first selected tag information, a second tag information selected from the tag information displayed on the display unit in accordance with the user's instructions, as the second selected tag information. The image extraction unit extracts the first search image to which the second selection tag information has been assigned, from among the first search images, as the second search image. The image display unit displays at least a portion of all second search images on the display unit. It is preferable that the tag information display unit displays at least a portion of all the tag information assigned to all second search images.

[0027] Furthermore, the tag information specification unit simultaneously specifies two or more tag information selected from the tag information displayed on the display unit in accordance with user instructions as selected tag information. The image extraction unit extracts images from the image group that have all of the two or more selected tag information attached as search images. The image display unit displays at least a portion of all searched images. It is preferable that the tag information display unit displays at least a portion of all the tag information assigned to all searched images.

[0028] Furthermore, the present invention includes the step of an instruction acquisition unit acquiring instructions input from a user, The image search unit searches for images from a group of images that match the search criteria entered in response to user instructions, and uses these images as search images. The image display unit performs the step of displaying the images included in the image group on the display unit, The image selection unit includes the step of selecting one image from among the images displayed on the display unit in accordance with instructions from the user, The search criteria are conditions for searching for images within a set of images based on criteria other than the time of capture. From the search images displayed on the screen, one search image selected according to the user's instructions is designated as the specified search image. The present invention provides an image processing method in which, when a specified search image is selected, only the specified search image is displayed on the display unit, and then, in response to user instructions, the specified search image and at least a portion of images included in the image group that were taken around the time the specified search image was taken are displayed on the display unit as a time-series image in chronological order.

[0029] Furthermore, the present invention provides a program for causing a computer to execute each of the above-described steps of the image processing method.

[0030] Furthermore, the present invention provides a computer-readable recording medium on which a program for causing a computer to execute each of the above-described steps of the image processing method is recorded.

[0031] Furthermore, the present invention includes an instruction acquisition unit that acquires instructions input by the user, An image search unit searches for images from a group of images that match the search criteria entered by the user, and uses them as search images. An image display unit that displays the images included in the image group on the display unit, The system includes an image selection unit that selects one image from among the images displayed on the display unit according to instructions from the user, The instruction acquisition unit, image search unit, image display unit, and image selection unit are hardware or a processor that executes a program. The search criteria are conditions for searching for images within a set of images based on criteria other than the time of capture. The image selection unit selects one search image from the search images displayed on the display unit according to the user's instructions and designates it as the specified search image. The image display unit, when a specified search image is selected, displays only the specified search image on the display unit, and then, in response to instructions from the user, displays the specified search image and at least a portion of images included in the image group that were taken around the time the specified search image was taken, as a time-series image on the display unit in chronological order.

[0032] Here, the image search unit, A tag information display unit that displays tag information assigned to images included in a group of images, A tag information specification unit that designates selected tag information as selected tag information from among the tag information displayed on the display unit, in accordance with instructions from the user, It comprises an image extraction unit that extracts images with selected tag information from a group of images as search images, The tag information display unit, tag information specification unit, and image extraction unit are hardware or a processor that executes a program. The tag information display unit displays at least a portion of all the tag information assigned to all images included in the image group. The tag information specification unit designates the first tag information selected from the tag information displayed on the display unit in accordance with the user's instructions as the first selected tag information. The image extraction unit extracts an image from the image group that has been assigned the first selection tag information as the first search image. The image display unit displays at least a portion of all first search images on the display unit. It is preferable that the tag information display unit displays at least a portion of all the tag information assigned to all first search images. [Effects of the Invention]

[0033] According to the present invention, in order to display images taken before and after the time a searched image was taken, the user does not need to return to the image list screen, but can view images taken before and after the time a specified searched image was taken in chronological order, in response to the user's instructions. [Brief explanation of the drawing]

[0034] [Figure 1] This is a block diagram of one embodiment showing the configuration of an image processing system according to the present invention. [Figure 2] This is a block diagram of one embodiment representing the client configuration. [Figure 3] This is a flowchart illustrating one embodiment of the client's behavior. [Figure 4] This is a conceptual diagram of one embodiment representing the home screen of an application displayed on the client's display unit. [Figure 5] This is a conceptual diagram of one embodiment representing the image viewing screen for a specified image. [Figure 6] This is a conceptual diagram of one embodiment representing a screen for viewing information about a specified image. [Figure 7] This is a conceptual diagram of one embodiment representing the editing screen for the information of a specified image. [Figure 8] This is a conceptual diagram of one embodiment representing a new tag information input screen. [Figure 9] This is a conceptual diagram of one embodiment representing a tag information selection screen. [Figure 10] This is a conceptual diagram of one embodiment showing the state in which the first tag information has been selected on the tag information selection screen. [Figure 11] This is a conceptual diagram of one embodiment showing how the content of the tag information is updated according to the first tag information on the tag information selection screen. [Figure 12] This is a conceptual diagram of one embodiment showing the state when the second tag information is selected on the tag information selection screen. [Figure 13] This is a conceptual diagram of one embodiment representing an image list screen. [Figure 14] This is a conceptual diagram of one embodiment showing how a search image (1) is selected on the image list screen. [Figure 15] This is a conceptual diagram of one embodiment representing the screen for viewing specified search images. [Figure 16] This is a conceptual diagram of one embodiment representing a time-series image list screen. [Figure 17] This is a conceptual diagram of one embodiment showing what happens when the "Search Tags" button is tapped on the tag information selection screen. [Figure 18] This is a conceptual diagram of one embodiment representing a search screen for tag information. [Figure 19] This is a conceptual diagram of one embodiment representing a tag information selection screen. [Figure 20]This is a conceptual diagram illustrating an example of a collage image created using several images selected from a user's collection of images dated "2015 / 01 / 10". [Figure 21] This is a conceptual diagram representing a group of images taken on the same day as the images used in the collage: "January 10, 2015 (Saturday)". [Figure 22] This is a conceptual diagram illustrating an example of an image that corresponds to the keyword "soccer." [Modes for carrying out the invention]

[0035] The image processing system, image processing method, program, and recording medium of the present invention will be described in detail below based on preferred embodiments shown in the attached drawings.

[0036] Figure 1 is a block diagram of one embodiment showing the configuration of an image processing system according to the present invention. The image processing system 10 shown in Figure 1 comprises a server 12 and a plurality of clients 14, each connected to the server 12 via a network 16.

[0037] Server 12 has functions such as acquiring image sets owned by each of the multiple users who use the image processing system 10, storing each image included in the image set owned by each user, and sharing those images only with the sharing partners specified by each user. Server 12 is not limited to one unit, but may consist of multiple units, and is composed of workstations, etc., which have control devices, storage devices, and communication devices.

[0038] Client 14 is an image processing device of the present invention, which has the function of searching for images from a group of images owned by the user based on search conditions, more specifically using tag information attached to the images as image search conditions, and displaying the resulting images on the display unit. Client 14 consists of a desktop PC (personal computer), notebook PC, tablet PC, or mobile terminal such as a mobile phone or smartphone, which has a control device, input device, storage device, communication device, and display. The following explanation will use the case where client 14 is using a smartphone as an example.

[0039] Figure 2 is a block diagram of one embodiment representing the configuration of a client. The client 14 shown in Figure 2 includes an instruction acquisition unit 18, an image holding unit 20, an image analysis unit 22, an evaluation value assignment unit 24, a tag information assignment unit 26, a weighting setting unit 28, a tag information display unit 30, a tag information specification unit 32, an image extraction unit 34, a tag information search unit 36, an image display unit 38, an image specification unit 40, an image information display unit 42, an image information editing unit 44, and a display unit 46.

[0040] In client 14, first, the instruction acquisition unit 18 acquires instructions input from the user of client 14.

[0041] Users can input various instructions by touching the touch panel of their smartphone. Instructions can also be entered using other input devices such as a mouse, keyboard, and trackpad. User instructions, as will be described in detail later, include instructions to select tag information, instructions to select an image, instructions to input tag information to be assigned to an image, instructions to input search conditions (keywords) for tag information, and instructions to display the image on the display unit 46.

[0042] Next, the image holding unit 20 is a storage device such as a semiconductor memory, and it holds the image group.

[0043] The image set consists of images taken by the user using the smartphone's camera and stored within the smartphone. The image set is not particularly limited and may include, for example, images imported into the smartphone from external sources, online images stored on social networking systems (SNS) and online storage, and combinations of these image sets.

[0044] For example, images taken with a smartphone camera can be configured to be automatically uploaded from the smartphone to the server 12. Images uploaded to the server 12 are stored in each user's storage area within the server 12's storage device. Images uploaded to the server 12 can also be configured to remain in the smartphone's image storage unit 20, or to be deleted. Furthermore, images such as those taken with a digital camera, images stored on a PC, notebook PC, tablet PC, etc., and images scanned from photo prints, etc. (digital image data) can be imported into the smartphone. Images imported into the smartphone from an external source are stored in the image storage unit 20 and treated as part of the image group.

[0045] Next, the image analysis unit 22 analyzes each image included in the image group.

[0046] The items to be analyzed in an image are not particularly limited, but include, for example, the detection of subjects in the image (person detection and object detection), the detection of people's faces, the detection of people's expressions, the detection of people's actions, the detection of scenes (nightscape, sea, beach, sky, etc.), the detection of events (sports day, wedding, graduation ceremony, etc.), and the detection of user preferences (hobbies), as well as the analysis of image quality such as brightness, hue, contrast, and degree of blur, and the analysis of time information (time of shooting) and location information (location of shooting) contained in additional image information such as Exif (Exchangeable Image File Format).

[0047] Next, the evaluation value assignment unit 24 assigns an evaluation value to each image based on the results of the analysis of each image.

[0048] The method by which the evaluation value assignment unit 24 assigns an evaluation value to an image is not particularly limited, but for example, it can assign a score to the result of each analysis of the image, calculate a total score from the scores of multiple analyses, and assign this total score as the evaluation value of the image. For example, images of people smiling will receive higher scores than images of people crying or angry. Similarly, images with appropriate brightness will receive higher scores than images that are too bright or too dark. Furthermore, the evaluation value assignment unit 24 sets a higher evaluation value for images to which tag information was selected later, in accordance with the weighting settings for the tag information described later, compared to images to which tag information was selected earlier in time.

[0049] Next, the tag information assignment unit 26 assigns tag information to each image based on the results of the analysis of each image.

[0050] Tag information consists of various pieces of information about an image, such as the date and location where the image was taken, the names of the subjects in the image, the names of the subjects' actions, names representing scenes, events, or emotions of people that can be gleaned from the image, and names representing the user's preferences. Tag information is text information, and it is possible to search for tag information using keywords. Tag information may include tag information for a higher-level concept that has tag information for multiple lower-level concepts. For example, it may include tag information for a higher-level concept, "sweets," which has tag information for two lower-level concepts, "Mont Blanc" and "strawberry shortcake." Furthermore, tag information may also include confidence level information, which indicates the reliability of the tag information. While automatically assigned tag information may not always be accurate, the accuracy of the tag information can be judged based on the confidence level information. The tag information assignment unit 26 is not a mandatory component, but it may assign additional tag information to an image that already has tag information assigned to it, or it may assign new tag information to an image that does not have tag information assigned to it.

[0051] Next, the weight setting unit 28 sets weights for the tag information.

[0052] The method by which the weighting setting unit 28 sets weights for tag information is not particularly limited, but when a user searches for an image using two or more tag pieces of information, it is likely that the tag pieces selected later are more important in terms of time than the tag pieces selected earlier. Accordingly, the weighting setting unit 28 can set a heavier weight for the tag pieces selected later in terms of time than for the tag pieces selected earlier among the two or more tag pieces of information simultaneously selected by the user in response to the tag information displayed on the display unit 46.

[0053] Next, the tag information display unit 30 displays the tag information assigned to the images included in the image group on the display unit 46.

[0054] The method by which the tag information display unit 30 displays tag information is not particularly limited, but for example, the tag information displayed on the display unit 46 can be classified and displayed in multiple categories. This allows the tag information to be classified and displayed according to its type, so that the user can easily find the tag information they want to use as a search criterion for images from among the tag information displayed on the display unit 46.

[0055] In this case, the tag information display unit 30 can display tag information included in at least one of multiple categories in descending order of the number of images to which the tag information is attached. This allows for the priority display of tag information attached to many images. Tag information represented by consecutive numbers, such as the date the image was taken, may be displayed in descending or ascending order of the numbers.

[0056] Furthermore, the tag information display unit 30 may display tag information included in at least one of multiple categories in order of the number of times it has been used as a keyword for searching for images, or, if the tag information has confidence level information, it may display the tag information included in at least one of multiple categories in order of the confidence level of the tag information, based on the confidence level information of the tag information.

[0057] Furthermore, the tag information display unit 30 can display the tag information of multiple subordinate concepts included in the tag information of a higher-level concept on the display unit 46 when the tag information of a higher-level concept is selected in response to instructions from the user. Furthermore, the tag information display unit 30 can display tag information as buttons (links) on the display unit (touch panel) 46. This allows users to easily select tag information.

[0058] Next, the tag information designation unit 32 designates the tag information selected from the tag information displayed on the display unit 46 in accordance with the user's instructions as the selected tag information. Furthermore, the tag information designation unit 32 designates the search tag information selected from the search tag information displayed on the display unit 46 in accordance with the user's instructions as the selected tag information.

[0059] Next, the image extraction unit 34 extracts images from the image group that have been assigned the selected tag information specified by the tag information specification unit 32 as search images.

[0060] Here, the tag information display unit 30, the tag information specification unit 32, and the image extraction unit 34 constitute the image search unit of the present invention, which searches for images from a group of images that match the search conditions entered in response to instructions from the user, and uses them as search images.

[0061] Next, the tag information search unit 36 ​​searches for tag information from among all the tag information assigned to all images included in the image group that matches, at least partially, the keyword entered in response to the user's instructions, and uses that information as search tag information.

[0062] Next, the image display unit 38 displays the images included in the image group on the display unit 46.

[0063] The image display unit 38 displays, for example, each image included in the image group, a list of images included in the image group (thumbnail images), a list of searched images (thumbnail images), and so on. When displaying a list of images or a list of searched images on the display unit 46, the image display unit 38 can display the images chronologically in the order of their shooting time. Furthermore, if each image in the image group has image evaluation value information, the image display unit 38 can also display the images on the display unit 46 in descending order of their image evaluation values.

[0064] Next, the image selection unit 40 selects one image from among the images displayed on the display unit 46, according to the user's instructions.

[0065] The user can select one image they wish to view from the list of images displayed on the display unit 46. If one image is selected, the image display unit 38 will display only this selected image on the display unit 46.

[0066] Next, the image information display unit 42 displays the information of one image specified by the image selection unit 40, that is, one image selected according to the user's instructions, on the display unit 46.

[0067] Image information includes various pieces of information about the image. In this embodiment, it includes the time the image was taken and tag information assigned to the image. However, it is not limited to this, and may also include the image file name, image size, resolution, location information, etc.

[0068] Next, the image information editing unit 44 edits the information of image 1 displayed on the display unit 46 by the image information display unit 42 in response to instructions from the user.

[0069] Users can edit the tag information assigned to images. The tag information automatically assigned by the tag information assignment unit 26 may contain inaccurate information. Users can correct or delete tag information that they believe to be inaccurate. Furthermore, users can also add new tag information to an image. In this case, the image information editing unit 44 adds the new tag information entered in response to the user's instructions to the image as information for the image 1 displayed on the display unit 46.

[0070] Next, the display unit 46 displays various types of information.

[0071] The display unit 46 displays images, image information, a list of images, a list of tag information, etc. The display unit 46 is a liquid crystal panel, an organic EL (Electro Luminescence) panel, etc., and in this embodiment, it is a touch panel.

[0072] Next, referring to the flowchart shown in Figure 3, we will explain the operation of client 14 when searching for a desired image from a group of images using tag information attached to the image as a search criterion.

[0073] First, let's explain what happens when the first tag information is selected. In this case, first, the tag information display unit 30 displays at least a portion of all the tag information assigned to all the images included in the image group on the display unit 46 (step S1). When the tag information is displayed on the display unit 46, the tag information designation unit 32 then designates the first tag information selected from the tag information displayed on the display unit 46 in accordance with the user's instructions as the first selected tag information (selected tag information) (step S2). Once the first selection tag information is specified, the image extraction unit 34 then extracts the image to which the first selection tag information is attached from the group of images as the first search image (search image) (step S3). Once the first search images are extracted, the image display unit 38 then displays at least a portion of all the first search images on the display unit 46 (step S4). Next, the tag information display unit 30 displays at least a portion of all the tag information assigned to all first search images on the display unit 46 (step S5).

[0074] Next, we will explain the behavior when the second tag information is selected simultaneously with the first tag information. In this case, the tag information designation unit 32 then simultaneously designates a second tag information selected from the tag information displayed on the display unit 46 in accordance with the user's instructions, in addition to the first selected tag information, as the second selected tag information (selected tag information) (step S6). Next, the image extraction unit 34 extracts the first search image to which the second selection tag information has been assigned as the second search image (search image) from among the first search images (step S7). Next, the image display unit 38 displays at least a portion of all of the second search images on the display unit 46 (step S8). Next, the tag information display unit 30 displays at least a portion of all the tag information assigned to all the second search images on the display unit 46 (step S9).

[0075] The behavior is the same when three or more tag pieces are selected simultaneously. In other words, the behavior when two or more tag pieces are selected simultaneously is as follows: First, the tag information designation unit 32 simultaneously designates two or more tag pieces of information selected from the tag information displayed on the display unit 46 in accordance with the user's instructions, as selected tag information. Next, the image extraction unit 34 extracts images from the image group that have all of the two or more selected tag information attached as search images. Next, the image display unit 38 displays at least a portion of all searched images on the display unit 46. Next, the tag information display unit 30 displays at least a portion of all the tag information assigned to all searched images on the display unit 46.

[0076] The image processing apparatus of the present invention will be described below using the case in which the client 14 is implemented by an application program (hereinafter also simply referred to as "app") running on a smartphone as an example.

[0077] Figure 4 is a conceptual diagram of one embodiment showing the home screen of an application displayed on the client's display unit. The smartphone display unit 46 shown in Figure 4 is composed of a touch panel, and by touching the touch panel, the user can use the display unit (touch panel) 46 not only to display various information but also as an instruction acquisition unit 18 that acquires various instructions input by the user.

[0078] At the top of the home screen, the notification and status area 50 is displayed, and below it, the menu area 52 is displayed. Additionally, at the bottom of the home screen, the navigation area 54 is displayed.

[0079] Within the navigation area 54, the following buttons are displayed in order from left to right: "Search," "Organize," "Home," "Share," and "Various." When the "Search" button is tapped (pressed), the tag information selection screen is displayed on the display unit 46. Additionally, when the "Combine" button is tapped, a composite image such as a photo album is automatically created using the images included in the image group. When the "Home" button is tapped, the home screen, that is, a list of images included in the image group, is displayed on the display unit 46. In the image list, the images are displayed in order, for example, by the time they were taken (descending or ascending order). When the "Share" button is tapped, the image selected according to the user's instructions is shared with other users. When the "Various" button is tapped, you can select and execute various other functions that are not displayed in the navigation area.

[0080] A list of images is displayed in the area between the menu area 52 and the navigation area 54. Before searching for images, when no tag information has been selected as a search criterion, the image list screen displays at least a portion of all the images included in the image list, for example, the image group, in chronological order from top to bottom, in descending order of the time the images were taken, in this embodiment.

[0081] As shown by the fingerprint 56 in Figure 4, the user can view only the image they want to see by tapping the image 1 displayed on the image list screen, and the tapped image 1 will be displayed on the display unit 46.

[0082] In this case, the image selection unit 40 selects one image from the list of images displayed on the display unit 46 according to the user's instructions and designates it as the designated image. The image display unit 38 then displays only the designated image on the display unit 46.

[0083] Figure 5 is a conceptual diagram of one embodiment showing the screen for viewing a specified image. The screen for viewing a specified image is displayed across the entire surface of the display unit 46. As shown in Figure 5, an "×" button 58 is displayed in the upper left of the screen for viewing a specified image, and an "Info" button 60 is displayed in the lower right. A "Trash" button 62 and a "Share" button 64 are displayed at the bottom of the screen for viewing a specified image.

[0084] For example, a user can close the image viewing screen and return to the image list screen (home screen) by tapping the "×" button 58. The user can also delete the image by tapping the "trash" button 62, or share the image with other users by tapping the "share" button 64. Furthermore, as shown by the fingerprint 56 in Figure 5, the user can flip the image over and view its information by tapping the "information" button 60.

[0085] Figure 6 is a conceptual diagram of one embodiment showing a screen for viewing information about a specified image. An "×" button 66 is displayed in the upper left of the screen for viewing information about a specified image, and an "Edit" button 68 is displayed in the upper right. In addition, a thumbnail image of the specified image is displayed at the top of the screen for viewing information about a specified image, and below it, the time the specified image was taken, automatically assigned tags, manually assigned tags, and an "Add Tag" button 70 are displayed in that order.

[0086] Automatically assigned tags are categorized into Calendar, People, Shooting Location, Automatically Assigned Tags, and Service Tags, while manually assigned tags are categorized as Manually Assigned Tags. The calendar category includes buttons for tag information such as "2018" and "December," indicating the year and month the specified image was taken. The "People" category includes a button with the tag information for "Taro". The shooting location category includes buttons for the tags "Park," "Minato Ward," and "Odaiba Seaside Park." The categories for automatically assigned tags include buttons for the tags "Outdoor," "Boy," and "Sunny." The scan tag category includes a button for "Scan Service" tag information. The categories for manually assigned tags include a button for the tag information "Kids' Fashion".

[0087] For example, a user can close the information viewing screen for a specified image and return to the same screen by tapping the "×" button 66. Additionally, a user can edit the information for a specified image by tapping the "Edit" button 68, as indicated by the fingerprint 56.

[0088] Figure 7 is a conceptual diagram of one embodiment showing the editing screen for the information of a specified image. In the editing screen for the information of a specified image shown in Figure 7, a "Cancel" button 72 is displayed in the upper left and a "Save" button 74 is displayed in the upper right.

[0089] When the "Edit" button 68 is tapped on the screen for viewing the information of the specified image, an "×" button is displayed in the buttons for tag information that can be deleted on the editing screen for the information of the specified image. When the user taps the "×" button in the buttons for tag information, the image information editing unit 44 deletes that tag information button. Also, as shown by the fingerprint 56 in Figure 7, when the user taps the "Add Tag" button 70, a screen for entering new tag information is displayed.

[0090] If the "Cancel" button 72 is tapped on the editing screen for the specified image, the edits made to the specified image will be discarded, and the user will return to the viewing screen for the specified image. If the "Save" button 74 is tapped, the edits made to the specified image will be saved.

[0091] Figure 8 is a conceptual diagram of one embodiment showing a screen for entering new tag information. At the top of the screen for entering new tag information shown in Figure 8, a thumbnail image of the specified image is displayed, and below it, a field 76 for entering new tag information is displayed. The field 76 for entering new tag information displays the message "Add Tag" to prompt the user to enter new tag information. To the right of the field 76 for entering new tag information, a "Cancel" button 78 is displayed.

[0092] When a user taps the input field 76 for new tag information, a history of recently assigned new tag information is displayed below the input field 76, and a software keyboard is displayed at the bottom of the new tag information input screen. The user can use the software keyboard to enter the name of the new tag information into the input field 76.

[0093] In this case, each time the user enters text into the new tag information input field 76, a partial match search is performed. Tag information that partially matches the text entered into the new tag information input field 76 is searched from all the tag information assigned to all the images in the image group and displayed on the new tag information input screen. The user can also select new tag information from the list of tag information displayed on the new tag information input screen.

[0094] If the user taps the "Cancel" button 78, the new tag information input screen will be closed, and the user will return to the editing screen for the specified image's information.

[0095] Next, we will explain the tag information selection screen. As mentioned above, when the "Search" button is tapped, the tag information selection screen is displayed on the display unit 46.

[0096] Figure 9 is a conceptual diagram of one embodiment representing the tag information selection screen. The tag information selection screen is displayed overlaid on the upper layer of the image list screen, except for a portion of the upper area, so that the list of images below is visible with a certain degree of transparency. When no tag information is selected, the tag information selection screen displays a portion of all the tag information assigned to all images in the image group, categorized into multiple categories. Each tag information is displayed as a button within the tag information selection screen. Also, a "Search Tags" button 80 is displayed at the bottom of the tag information selection screen.

[0097] The multiple categories include calendar, people, shooting location, automatically assigned tags, manually assigned tags, and service tags. The calendar category includes buttons for tag information indicating the year and month the image was taken. The "People" category includes buttons with tag information for the names of the people pictured in the image. The "Shooting Location" category includes buttons with tag information for the location where the image was taken. The categories of automatically assigned tags include buttons with tag information automatically assigned by the tag information assignment unit 26. The category of manually assigned tags includes buttons for tag information manually assigned by users, i.e., buttons for tag information assigned by the image information editing department 44. The service tag categories include buttons for tag information related to services provided by the image processing system 10 according to the present invention, such as scanning prints to create digital image data.

[0098] On the tag information selection screen, the calendar category displays the tag information buttons for "2018," "2017," and "2016" as the year the image was taken, in that order from left to right, and the tag information buttons for "December," "November," "October," and "September" as the month the image was taken, in that order from left to right. In the People category, the tag information buttons for the names of the people, "Takashi," "Akira," and "Fuji Hanako," are displayed in that order from left to right. The shooting location category does not include tag information. In the automatically assigned tag categories, the tag information buttons for "bicycle," "ramen," and "drinking party" are displayed in that order from left to right. In the categories for manually assigned tags, the tag information buttons for "Circle," "Today's Lunch," and "Spectacular View" are displayed in that order from left to right. The service tag category displays a button for the tag information "Scanned Images".

[0099] The tag information buttons for each category other than the calendar category are displayed in a single horizontal row. For the calendar category, the tag information buttons for the year and month the image was taken are displayed in a single horizontal row, sorted in descending order of the year and month. The tag information buttons for each other category are displayed in descending order of the number of images to which each tag information is assigned.

[0100] Furthermore, if a category contains more tag information than can be displayed horizontally, users can tap the tag information button for that category and swipe left or right to push the displayed tag information buttons outwards and move any tag information buttons not currently displayed into the selection screen.

[0101] Users do not need to type in keywords to search for images; they can simply tap and select the tag information corresponding to the desired image from the tag information displayed on the display unit 46, allowing for an intuitive search.

[0102] Here, let's assume that, with no tag information selected as a search criterion, the user taps the "bicycle" tag information button as the first tag information from the tag information buttons displayed on the tag information selection screen, as shown by the fingerprint 56 in Figure 10.

[0103] In this case, the tag information specification unit 32 specifies the tag information for "bicycle" corresponding to the button for the "bicycle" tag selected by the user as the first selected tag information. In addition, the tag information display unit 30 changes the color of the button for the "bicycle" tag information and displays "bicycle" on the left side of the upper area of ​​the image list screen that is not covered by the tag information selection screen.

[0104] Next, the image extraction unit 34 extracts images from the image group that have the tag information "bicycle," which is the first selected tag information, as the first search images, and the image display unit 38 displays at least a portion of all of the first search images on the image list screen.

[0105] Furthermore, as shown in Figure 11, the tag information display unit 30 displays at least a portion of all the tag information assigned to all first search images on the tag information selection screen. In other words, the content of the tag information displayed on the tag information selection screen is updated.

[0106] In this embodiment, the calendar categories are changed from buttons with tag information for "2018" and "2017" to buttons with tag information for "2018" and "2017" as the year the image was taken. Additionally, the character categories will be changed from buttons labeled "Takashi," "Akira," and "Fuji Hanako" to buttons labeled "Takashi," "Akira," and "Yamada-san." The categories for automatically assigned tags will be changed from "Bicycle," "Ramen," and "Drinking Party" to buttons for tag information related to "Bicycle," "Mountain," and "Guardrail." The categories for manually assigned tags will be changed from "Circle," "Today's Lunch," and "Spectacular View" to buttons for tag information for "Circle," "Spectacular View," and "Cycling Trip."

[0107] In other words, if only the "bicycle" tag information button is selected from the tag information buttons displayed on the tag information selection screen, the image group will be narrowed down to only the search images that have the "bicycle" tag information, and further narrowed down from all the tag information attached to all the images in the image group to only the tag information attached to all the search images that have the "bicycle" tag information.

[0108] Next, as shown by the fingerprint 56 in Figure 12, suppose the user taps the "bicycle" tag information button as the second tag information from the tag information buttons displayed on the tag information selection screen, in addition to the first tag information button, "bicycle trip". In other words, with the "bicycle" tag information button selected, the "bicycle trip" tag information button is also selected, so two tag information items are selected simultaneously.

[0109] In this case, the tag information specification unit 32 specifies, in addition to the first tag information for "bicycle," the second tag information for "bike trip," which corresponds to the button for the "bike trip" tag selected according to the user's instructions, as the second selected tag information. In other words, both the "bicycle" and "bike trip" tag information are specified simultaneously as selected tag information. Furthermore, the tag information display unit 30 changes the color of the button for the "bike trip" tag information in addition to the button for the "bicycle" tag information, and displays "bike trip" to the right of "bicycle" in the upper area of ​​the image list screen that is not covered by the tag information selection screen.

[0110] Next, the image extraction unit 34 extracts the first search image tagged with "bicycle" from the first search images tagged with "bicycle trip" as the second search image, and as shown in Figure 13, the image display unit 38 displays at least a portion of the second search image on the image list screen. Although not shown in the illustration, the tag information display unit 30 displays at least a portion of all tag information assigned to all second search images on the tag information selection screen.

[0111] In other words, if both the "bicycle" and "bike trip" tag buttons are selected simultaneously, the search results will be filtered to show only the images tagged with both "bicycle" and "bike trip" from the image group, and further filtered to show only the tags attached to all search images tagged with both "bicycle" and "bike trip" from all the tags attached to all the images in the image group.

[0112] Users can close the tag information selection screen at any time by swiping down the downward arrow 92 displayed at the top of the tag information selection screen. When the tag information selection screen is closed, the image list screen for the area previously covered by the tag information selection screen is displayed, as shown in Figure 13. This allows the user to view a list of searched images.

[0113] If the image list screen contains more search images than can be displayed, the user can swipe the displayed search images up or down to move them, pushing some of the displayed search images outwards and moving some of the images not currently displayed into the image list screen for display.

[0114] Furthermore, when the tag information selection screen is closed, an upward-pointing arrow 94 is displayed at the top of the tag information selection screen, as shown in Figure 13. The user can open the tag information selection screen at any time by swiping this upward-pointing arrow 94 upwards. In this embodiment, the tag information selection screen can be switched between three states: the fully open state shown in Figures 9-12, the fully closed state shown in Figure 13, and an intermediate position. Note that the tag information selection screen may also be switched between more than three states.

[0115] Furthermore, users can access the tag selection screen at any time by tapping the "Search" button while the image list screen is displayed. Alternatively, users can further refine their image search by selecting additional tag buttons when the two tag categories "Bicycle" and "Bike Trip" are selected, thereby narrowing down the tag information even further.

[0116] The two tag information items, "bicycle" and "bike trip," which are designated as selected tag information, are maintained as selected tag information by the tag information designation unit 32 until they are removed from the designation as selected tag information in response to instructions from the user. For example, a "Deselect" button could be provided on the tag information selection screen, allowing the user to deselect both "bicycle" and "bike trip" tags at once by tapping the "Deselect" button.

[0117] Furthermore, users can toggle between selecting and deselecting a tag by tapping the tag information button. In this case, the tag information specification unit 32 switches between designating the tag information corresponding to the button as selected tag information and de-designating the tag information corresponding to the button as selected tag information each time a button is pressed, in response to instructions from the user.

[0118] In other words, the selection of tag information is maintained until it is deselected. Therefore, if the selection of tag information is maintained, that is, if an image search is performed using the selected tag information, and then the user switches to another display screen and returns to the tag information selection screen, the searched images resulting from the search using the selected tag information can be viewed at any time.

[0119] Furthermore, when the tag information is displayed as a button on the display unit 46, the user can select a higher-level tag information by long-pressing (long-tapping) the button for tag information 1.

[0120] In this case, if a button for one tag information selected by the user from among the tag information buttons displayed on the display unit 46 is pressed for a certain period of time or longer, the tag information designation unit 32 designates all subordinate tag information included in the higher-level tag information corresponding to the pressed tag information button as selected tag information. For example, if the button for the "Mont Blanc" tag information is long-pressed, all "sweets" tag information, which is a higher-level concept than "Mont Blanc," will be selected.

[0121] Alternatively, the tag information designation unit 32 may, when one tag information is selected from the tag information displayed on the display unit 46 in response to instructions from the user, deselect any other tag information that has already been designated, and designate only the selected tag information as the selected tag information.

[0122] Next, as shown by the fingerprint 56 in Figure 14, the user can tap on the search image they want to view from the list of search images displayed on the image list screen, thereby displaying and viewing only the tapped search image on the display unit 46.

[0123] In this case, the image selection unit 40 selects one search image from among the search images displayed on the display unit 46 according to the user's instructions and designates it as the designated search image, and the image display unit 38 displays only the designated search image on the display unit 46.

[0124] Figure 15 is a conceptual diagram of one embodiment showing the screen for viewing specified search images. The screen for viewing specified search images is displayed across the entire surface of the display unit 46. In the upper left of the screen for viewing specified search images shown in Figure 15, an "×" button 58 is displayed, in the lower left, a "Calendar" button 82 is displayed, and in the lower right, an "Information" button 60 is displayed. At the bottom of the screen for viewing specified search images, a "Trash" button 62, a "Share" button 64, and an "Other" button 84 are displayed. The screen for viewing the specified search image has the same configuration as the screen for viewing the specified image, except that a "Calendar" button (82) is displayed in the lower left corner.

[0125] For example, a user can close the viewing screen for the specified searched image and return to the list of searched images by tapping the "×" button 58. Users can also delete the specified searched image by tapping the "trash" button 62, share the specified searched image with other users by tapping the "share" button 64, or perform other functions by tapping the "other" button 84. Furthermore, users can view information about the specified searched image by tapping the "information" button 60.

[0126] Additionally, users can view search results by displaying them one by one in sequence. In this case, after only the specified search image is displayed on the display unit 46, the image display unit 38, each time a user inputs an instruction, displays on the display unit 46 only one search image from all the search images that was taken before or after the time the specified search image was taken, in chronological order from the time the specified search image was taken.

[0127] For example, if the user swipes a specified search image to the right, only the first search image taken immediately before the specified image will be displayed from all the search images. From then on, each time the user swipes a search image to the right, the first search image taken immediately before the specified image will be displayed in chronological order.

[0128] On the other hand, if the user swipes the specified search image to the left, only the first search image taken immediately after the specified image will be displayed from all the search images. From this point onward, each time the user swipes the search image to the left, only the first search image taken immediately after the specified image will be displayed sequentially, in chronological order.

[0129] Furthermore, as shown in Figure 15, users can tap the "Calendar" button 82 displayed on the image viewing screen to display and view images taken around the time of the specified search image on the display unit 46 from the image group. In this case, when a specified search image is selected, only the specified search image is displayed on the display unit 46. Then, in response to user instructions, the image display unit 38 displays the specified search image and at least a portion of the images included in the image group that were taken around the time the specified search image was taken, as a time-series image on the display unit 46 in chronological order.

[0130] Thus, in order for the user to view images taken before and after the time the searched image was taken, the user does not need to return to the image list screen such as the home screen. Instead, in response to the user's instruction, in this embodiment, the user can view images taken before and after the time the specified searched image was taken in chronological order with a single action, such as tapping the "Calendar" button 82.

[0131] Here, the same thing could be achieved by, for example, returning to an image list screen such as the home screen in response to user instructions, and displaying images taken around the time the specified search image was taken, in chronological order. However, with this method of operation, if a large number of images spanning several years are stored on the smartphone, the user would be forced to navigate through several years' worth of images on the image list screen such as the home screen until the day the desired image was taken.

[0132] In contrast, by using the image processing device of this embodiment, it becomes possible to view images taken before and after the time the specified search image was taken in chronological order with a single action of tapping the "Calendar" button 82. This is extremely convenient compared to using conventional technology, which, as described above, requires returning to the image list screen such as the home screen to view images taken before and after the time the specified search image was taken in chronological order.

[0133] Figure 16 is a conceptual diagram of one embodiment representing a time-series image list screen. The time-series image list screen is displayed in the area between the menu area 52 and the navigation area 54.

[0134] Similarly, a user can view only one time-series image by tapping it on the time-series image list screen, thereby displaying only that tapped time-series image on the display unit 46. In this case, the image selection unit 40 selects one time-series image from among the time-series images displayed on the display unit 46 according to the user's instructions and designates it as the designated time-series image. The image display unit 38 then displays only the designated time-series image on the display unit 46.

[0135] Furthermore, when a specified time-series image is selected, only the specified time-series image is displayed on the display unit 46. Then, each time a user inputs an instruction via the image display unit 38, only one image from the image group that was taken before or after the time of the specified time-series image is displayed on the display unit 46 in the order of images taken in chronological order from the time of the specified time-series image. The behavior when a specified time-series image is swiped to the right or left is the same as the behavior when a specified search image is swiped to the right or left.

[0136] Alternatively, instead of providing a dedicated "Calendar" button 82, a specific touch operation may be used to achieve the same functionality as the "Calendar" button 82.

[0137] Users can return to the search image list screen by tapping the "Home" button on the time-series image list screen. As mentioned above, the selection of tag information is maintained until it is deselected. Therefore, when you return to the search image list screen, the list of search images that are the results of the search using the selected tag information will be displayed on the search image list screen.

[0138] Furthermore, the search conditions for searching for an image from the image group before displaying the time-series images on the display unit 46 are not particularly limited, as long as they are conditions for searching for an image from the image group based on conditions other than the time the image was taken. The conditions for the time of image capture include not only the time the image was taken, but also various conditions related to the time the image was taken, such as the year, month, week, day, and time the image was taken. Conditions other than the time the image was taken include all conditions other than those related to the time the image was taken. The search criteria can include the image analysis results from the image analysis unit 22 and the names of the tag information assigned by the tag information assignment unit 26. Examples of search criteria include the names of subjects in the image (person names and object names), names of people's facial expressions, names of people's actions, names of people's emotions, scene names, event names, user preference names, and shooting location names.

[0139] Furthermore, if a user cannot find the button for the tag information they wish to use as a search criterion on the tag information selection screen, they can search for tag information by entering keywords, as shown by the fingerprint 56 in Figure 17, by tapping the "Search Tags" button 80.

[0140] When the "Search Tags" button 80 is tapped, the tag information search screen is displayed on the display unit 46, as shown in Figure 18.

[0141] The tag information search screen is displayed overlaid on the upper layer of the image list screen, except for a small area at the top, with the list of searched images below visible with a certain degree of transparency. At the top of the tag information search screen, a keyword input field 86 is displayed. The keyword input field 86 displays the message "Search Tags" to prompt the user to enter keywords. An "×" button 88 is displayed at the far right of the keyword input field 86, and a "Cancel" button 90 is displayed to the right of the keyword input field 86.

[0142] Next, let's assume the user has entered a keyword into the keyword input field 86.

[0143] In this case, the tag information search unit 36 ​​searches for tag information from among all the tag information attached to all images included in the image group that matches, at least partially, the keyword entered by the user in response to their instructions, and uses that information as search tag information. Next, the tag information display unit 30 displays at least a portion of the search tag information on the display unit 46. Next, the tag information specification unit 32 selects one search tag from the search tag information displayed on the display unit 46 according to the user's instructions. Next, the image extraction unit 34 extracts images from the image group that have been assigned the search tag information 1 as search images. Next, the image display unit 38 displays at least a portion of all searched images on the display unit 46. Next, the tag information display unit 30 displays at least a portion of all the tag information assigned to all searched images on the display unit 46.

[0144] For example, as shown in Figure 18, if "あ" is entered in the keyword input field 86 to search for tag information for "ice cream," a partial match search is performed, and tag information that partially matches "あ" is searched from all the tag information attached to all the images in the image group and used as the search tag information.

[0145] Next, as suggested search tag information, as shown in Figure 18, on the tag information search screen, for example, the buttons for the tags "Ai-chan," "Aa-chan," "Amaou," and "Ice Cream" are displayed in this order from top to bottom.

[0146] If the tag information search screen contains more search tags than can be displayed, the user can swipe up or down to move the list of suggested search tags, thereby pushing some of the displayed search tags outwards and moving some of the undisplayed search tags into the search screen for display.

[0147] Additionally, users can delete all keywords entered in the input field 86 by tapping the "×" button 88. They can also close the tag information search screen and return to the tag information selection screen by tapping the "Cancel" button 90. Furthermore, users can close the tag information search screen at any time by swiping the downward arrow 96 displayed at the top of the tag information search screen.

[0148] Next, let's assume that the user taps the button for the tag information "ice cream" from among the candidate buttons for search tag information displayed on the tag information search screen, as shown by the fingerprint 56 in Figure 18.

[0149] In this case, as shown in Figure 19, the tag information selection screen is displayed on the display unit 46. The subsequent operations are the same as when the "Bicycle" tag information button is selected while no tag information is selected.

[0150] In other words, the tag information for "ice cream" is specified as the selected tag information, and as shown in Figure 19, "ice cream" is displayed on the left side of the upper area of ​​the image list screen that is not covered by the tag information selection screen. Next, from the group of images, images tagged with the selected tag information "ice cream" are extracted as search images, and these search images are displayed on the image list screen. Additionally, all tag information attached to search images that have the selected tag information "ice cream" will be displayed on the tag information selection screen.

[0151] The subsequent actions are the same as when the "bicycle" tag information button is selected and then the "bike trip" tag information button is tapped in addition, allowing the user to select additional tag information buttons to use as search criteria.

[0152] Furthermore, if a user takes a new image, adds new tag information to an image, or deletes existing tag information from an image, the reliability of the tag information may be improved using machine learning, such as AI (Artificial Intelligence), based on these new images, new tag information, and deleted tag information.

[0153] Alternatively, the server 12 may be configured to have at least some of the functions that the client 14 has, such as image analysis, assigning evaluation values, assigning tag information, setting weights, extracting images, and searching for tag information, and the server 12 may perform at least some of these functions.

[0154] In the apparatus of the present invention, the hardware configuration of the processing unit (Processing Unit) that performs various processes such as the instruction acquisition unit 18, image analysis unit 22, evaluation value assignment unit 24, tag information assignment unit 26, weighting setting unit 28, tag information display unit 30, tag information specification unit 32, image extraction unit 34, tag information search unit 36, image display unit 38, image specification unit 40, image information display unit 42, and image information editing unit 44 may be dedicated hardware, or it may be various processors or computers that execute programs.

[0155] Various types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute software (programs) and function as various processing units; Programmable Logic Devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing; and Dedicated Electrical Circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed for performing particular processing.

[0156] A single processing unit may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs, or a combination of an FPGA and a CPU. Furthermore, multiple processing units may be composed of one of these various processors, or two or more of the multiple processing units may be combined and composed of a single processor.

[0157] For example, as exemplified by servers and client computers, one processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Alternatively, as exemplified by System on Chip (SoC), a processor is used that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip.

[0158] Furthermore, the hardware configuration of these various processors is, more specifically, an electrical circuit (Circuitry) made up of circuit elements such as semiconductor devices.

[0159] Furthermore, the method of the present invention can be implemented, for example, by a program that causes a computer to execute each of its steps. A computer-readable recording medium on which this program is recorded can also be provided.

[0160] Although the present invention has been described in detail above, the present invention is not limited to the above embodiments, and various improvements and modifications may be made without departing from the spirit of the present invention. [Explanation of symbols]

[0161] 10 Image Processing Systems 12 servers 14 Clients 16 Networks 18 Instruction acquisition part 20 Image holding unit 22 Image Analysis Department 24 Evaluation Value Assignment Section 26. Tag Information Assignment Section 28 Weighting setting section 30 Tag information display section 32 Tag Information Specification Section 34 Image extraction unit 36 Tag Information Search Section 38 Image display section 40 Image selection section 42 Image information display unit 44 Image Information Editorial Department 46 Display section 50 Notification and Status Area 52 Menu Area 54 Navigation area 56 Fingerprint 58, 66, 88 "×" button 60. The "Information" button 62. The "Trash" button 64. The "Share" button 68. The "Edit" button 70. The "Add Tag" button 72, 78, 90 "Cancel" button 74. The "Save" button 76 Input field for new tag information 80. The "Search Tags" button 82. The "Calendar" button 84. The "Other" button 86 Keyword input field 92, 94, 96 Arrows

Claims

1. An instruction acquisition unit that acquires instructions input from the user, An image display unit that displays the images included in the image group on the display unit, An image search unit searches for images from the aforementioned group of images that have tag information that matches the tag information, which is a search condition entered in response to instructions from the user, and uses these images as search images. The system includes an image selection unit that selects one search image from the search images displayed on the display unit in accordance with the user's instructions and designates it as the specified search image. The aforementioned tag information is information for searching for images from the aforementioned group of images using tag information other than the time of capture. The image display unit, when the specified search image is specified, displays only the specified search image on the display unit, and then, each time an instruction is input from the user, displays only one image from the group of images that was taken before or after the time the specified search image was taken, in chronological order from the time the specified search image was taken. This is an image processing system.

2. The aforementioned image search unit, A tag information display unit that displays tag information assigned to images included in the aforementioned group of images on the display unit, A tag information designation unit that designates selected tag information as selected tag information from among the tag information displayed on the display unit, in accordance with the user's instructions, The system includes an image extraction unit that extracts images to which the selected tag information has been assigned as search images from the aforementioned group of images, The tag information display unit displays at least a portion of all the tag information assigned to all images included in the image group. The tag information designation unit designates the first tag information selected from the tag information displayed on the display unit in accordance with the user's instructions as the first selected tag information. The image extraction unit extracts the image to which the first selection tag information is assigned from the group of images as the first search image. The image display unit causes at least a portion of all of the first search images to be displayed on the display unit. The image processing system according to claim 1, wherein the tag information display unit causes at least a portion of all tag information assigned to all of the first search images to be displayed on the display unit.

3. The tag information designation unit simultaneously designates, in addition to the first selected tag information, a second tag information selected from the tag information displayed on the display unit in accordance with the user's instructions, as the second selected tag information. The image extraction unit extracts the first search image to which the second selection tag information has been assigned from the first search image as the second search image. The image display unit causes at least a portion of all of the second search images to be displayed on the display unit. The image processing system according to claim 2, wherein the tag information display unit causes at least a portion of all tag information assigned to all of the second search images to be displayed on the display unit.

4. The instruction acquisition unit acquires instructions input from the user, The image display unit performs the step of displaying the images included in the image group on the display unit, The image search unit searches the group of images for images that have tag information matching the tag information, which is a search condition entered in response to instructions from the user, and uses these images as search images. The image selection unit includes the step of selecting one search image from among the search images displayed on the display unit in accordance with the user's instructions and designating it as the designated search image. The aforementioned tag information is information for searching for images from the aforementioned group of images using tag information other than the time of capture. An image processing method in which, when the specified search image is specified, only the specified search image is displayed on the display unit, and then, each time an instruction is input from the user, only one image from the group of images that was taken before or after the time the specified search image was taken is sequentially displayed on the display unit in chronological order from the time the specified search image was taken.

5. A program for causing a computer to perform each step of the image processing method described in claim 4.

6. A computer-readable recording medium on which a program for causing a computer to perform each step of the image processing method described in claim 4 is recorded.

Images