Image processing system, image processing method, program and recording medium

Abstract

To provide an image processing system, an image processing method, a program and a recording medium that can display in chronological order images taken before and after the shooting time of one image, from the one image in a list of images searched for based on conditions other than the shooting time of the image.SOLUTION: In an image processing system, an image processing method, a program and a recording medium of the present invention, a search condition is a condition for searching for an image from a group of images based on conditions other than shooting time. After an image display unit displays only a search image on a display unit, in response to an instruction by single action from a user, the image display unit displays the search image and at least some of images taken at a shooting date before and after the shooting date of the search image included in the image group as time-series images on the display unit in chronological order. The instruction by single action from the user is input by pressing a button displayed on a viewing screen of the search image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing system, an image processing method, a program, and a recording medium for searching for images based on search conditions from among a group of images and displaying the images resulting from the search on a display unit. [Background technology]

[0002] For example, when a user searches for a desired image from among a group of images stored in a smartphone, the user may display a list of images (thumbnail images) included in the group of images in chronological order on a display unit, and search for the desired image while going back through the list of images in chronological order. Also known is a method of inputting a keyword for searching for an image from among a group of images, and searching for an image that matches the keyword by using additional information of the image, such as tag information.

[0003] However, by displaying a list of images and going back through them, it is very difficult to find a desired image from among the large number of images contained in the image group. Also, by inputting a keyword to search for an image, it may be possible to properly search for a desired image if the keyword to be used is clear, but it is very difficult to properly search for a desired image if the keyword is not clear.

[0004] Prior art documents related to the present invention include Patent Documents 1 and 2.

[0005] Patent Document 1 describes a method for searching image data by performing a character string search based on additional information added to the image data. For example, it describes a method for searching either an AND / OR combination of an Exif character string to be searched for and a sticky note character string to be searched for, searching for image data having additional information matching the input character string, and displaying the image data corresponding to the search result.

[0006] Patent Document 2 describes a method in which a label displays a plurality of attributes and attribute values ​​associated with a digital image, and a user selects one of the attribute values ​​displayed in the label to select another collection of images, etc. It also describes a method in which, while viewing a selected collection of images, a user searches for all images of a particular person, displays those images in chronological order, and displays their associated labels. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2009-124270 A [Patent Document 2] Patent No. 5216591 Summary of the Invention [Problem to be solved by the invention]

[0008] Patent Document 1 describes performing an AND search of multiple strings of additional information, but it requires the user to input the strings of characters he or she wants to search for, making it difficult to intuitively search for images. Furthermore, Patent Document 2 describes the ability to view a collection of other images by selecting one of the attribute values ​​of a single digital image, but this only allows viewing of a collection of images that correspond to one attribute value, and it is not possible to narrow down images by, for example, multiple attribute values.

[0009] Furthermore, some conventional online photo services have a function of sending a notification to a user urging the user to view images, and urging the user to view past images that the user owns.

[0010] After a notification encouraging the user to view the images is sent to the user, when the user opens the notification, a collage image created using several images selected from the group of images owned by the user, for example, "2015 / 01 / 10," is displayed on the display unit, as shown in Figure 20.

[0011] Next, when the user presses the "Show more photos" button displayed below the collage image, a group of images taken on the same date as the images used in the collage image, "Saturday, January 10, 2015," are displayed on the display in descending order of the image's shooting time, as shown in Figure 21.

[0012] Furthermore, by moving the group of images displayed on the display unit upward or downward, the user can cause the group of images taken on the previous or following shooting dates to be displayed sequentially in the order of the shooting times of the images. For example, when the group of images displayed on the display unit is moved upward, the group of images taken on the most recent date before "Saturday, January 10, 2015", for example "Friday, January 9, 2015", is displayed on the display unit. On the other hand, when the group of images displayed on the display unit is moved downward, the group of images taken on the most recent date after "Saturday, January 10, 2015", for example "Sunday, January 18, 2015" is displayed on the display unit.

[0013] As such, conventional online photo services have a function that allows you to click on the "Show more photos" button from a collage image to display a group of images taken on the same date as the images used in the collage image in chronological order.

[0014] The collage image included in the notification encouraging viewing of images was created using several images selected from a group of images taken on "Saturday, January 10, 2015" through a search criterion related to the image shooting time, "Saturday, January 10, 2015." Therefore, from the collage image, it is possible to display in chronological order a group of images taken on "Saturday, January 10, 2015" that were taken before or after the shooting time of the images used in the collage image. However, conventional online photo services do not have a function to display, in chronological order, images taken before and after the shooting time of one of the images found by searching for images based on any criteria other than the shooting time of the image.

[0015] Furthermore, in conventional online photo services, when searching for an image matching a keyword from among a group of images, the group of images matching the keyword are displayed on the display unit in descending order of the image shooting time for each shooting date. For example, if you search for images matching a keyword such as "soccer" from among the images, nine images matching soccer will be found from the image group for "Saturday, December 5, 2015," as shown in Figure 22.

[0016] If the "down arrow" button displayed to the right of "Saturday, December 5, 2015" is pressed, all images in the "Saturday, December 5, 2015" image group, including images unrelated to soccer, are displayed, and the "down arrow" button changes to an "up arrow" button. If the "up arrow" button is pressed, the nine images related to soccer are displayed again, and the button returns to the "down arrow" button.

[0017] As shown in Figure 22, when searching for the keyword "soccer," images matching "soccer" were found on the shooting date "November 28, 2015," before images matching "soccer" were found on "Saturday, December 5, 2015."

[0018] Now, if we return to the list screen of chronological images, in which all images in the image group are displayed in order of the time they were taken, and look at the image group for "Saturday, December 5th, 2015," in the list of chronological images, the shooting date of the image next to "Saturday, December 5th, 2015" is "Sunday, December 6th, 2015," and the shooting date of the image before "Saturday, December 5th, 2015" is "Friday, December 4th, 2015."

[0019] In this way, conventional online photo services have a function that, when searching for images using the keyword "soccer" from among a group of images, displays images taken on dates on which there are images that match "soccer" by the date on which they were taken.

[0020] However, conventional online photo services do not have a function that allows a user to display, in chronological order, images taken before and after the shooting time of one of the images in a list of images taken on a hit shooting date, with a single action such as pressing a button. In other words, the user must remember the shooting date that the user wants to display in the list of chronological images, display the list of chronological images, and move to the shooting date that the user wants to display.

[0021] An object of the present invention is to provide an image processing system, an image processing method, a program, and a recording medium that can chronologically display, for one image in a list of images searched based on conditions other than the image's shooting time, images taken before and after the image's shooting time. [Means for solving the problem]

[0022] In order to achieve the above object, the present invention provides a method for controlling a user's input of an instruction, comprising: an image search unit that searches the image group for an image that matches a search condition input in response to an instruction from a user, as a search image; an image display unit that displays images included in the image group on a display unit; an image designation unit that designates one image selected in response to an instruction from a user from among the images displayed on the display unit; The search conditions are conditions for searching for images from among the image group based on conditions other than the shooting time, The image designation unit designates one search image selected in response to an instruction from the user from among the search images displayed on the display unit as a designated search image; The image display unit, when a specified search image is designated, displays only the specified search image on the display unit, and then, in response to an instruction from a user, displays the specified search image and at least a portion of images included in an image group that were taken before or after the shooting time of the specified search image in chronological order as time-series images on the display unit.

[0023] Here, it is preferable that the image display unit displays only the specified search image on the display unit, and then, each time an instruction is input from the user, displays on the display unit only one search image from among the search images that was taken before or after the shooting time of the specified search image, in the order of the search images taken in chronological order from the shooting time of the specified search image.

[0024] The image designation unit designates one time-series image selected in response to an instruction from the user from among the time-series images displayed on the display unit as a designated time-series image; When a specified time-series image is specified, it is preferable that the image display unit displays only the specified time-series image on the display unit, and then, each time an instruction is input from the user, displays only one image from the group of images that was taken before or after the shooting time of the specified time-series image on the display unit in the order in which the images were taken in chronological order from the shooting time of the specified time-series image.

[0025] In addition, the image search unit a tag information display unit that displays tag information assigned to images included in the image group on a display unit; a tag information designation unit that designates tag information selected from the tag information displayed on the display unit in response to an instruction from a user as selected tag information; an image extraction unit that extracts, from a group of images, an image to which the selected tag information has been added as a search image; the tag information display unit displays at least a part of all the tag information assigned to all the images included in the image group on the display unit; the tag information designation unit designates, as a first selected tag information, a first piece of tag information selected in response to an instruction from the user from among the tag information displayed on the display unit; The image extraction unit extracts an image to which the first selection tag information is added as a first search image from the image group; The image display unit displays at least a part of all the first search images on the display unit; It is preferable that the tag information display unit displays at least a part of all the tag information assigned to all the first search images on the display unit.

[0026] Furthermore, the tag information designation unit simultaneously designates, in addition to the first selected tag information, second tag information selected from the tag information displayed on the display unit in response to an instruction from the user as second selected tag information; the image extraction unit extracts, from the first search images, the first search images to which the second selection tag information has been added as second search images; The image display unit displays at least a part of all the second search images on the display unit; It is preferable that the tag information display section displays at least a part of all the tag information assigned to all the second search images on the display section.

[0027] The tag information designation unit simultaneously designates two or more pieces of tag information selected in response to an instruction from the user from among the tag information displayed on the display unit as selected tag information; The image extraction unit extracts, from the group of images, an image to which all of the two or more pieces of selected tag information have been added as a search image; The image display unit displays at least a part of all the searched images on the display unit; It is preferable that the tag information display unit displays at least a part of all the tag information assigned to all the search images on the display unit.

[0028] Further, the present invention provides a method for controlling a display device, comprising: An image search unit searches for an image matching a search condition inputted in response to an instruction from a user from among the images as a search image; A step of causing an image display unit to display images included in the image group on the display unit; The image designation unit designates one image selected in response to an instruction from a user from among the images displayed on the display unit; The search conditions are conditions for searching for images from among the image group based on conditions other than the shooting time, designating one search image selected in response to an instruction from a user from among the search images displayed on the display unit as a designated search image; To provide an image processing method for, when a specified search image is specified, displaying only the specified search image on a display unit, and then displaying the specified search image and at least some of images included in an image group that were photographed before and after the photographing time of the specified search image in chronological order as time-series images on the display unit in response to an instruction from a user.

[0029] The present invention also provides a program for causing a computer to execute each step of the image processing method described above.

[0030] The present invention also provides a computer-readable recording medium having recorded thereon a program for causing a computer to execute each step of the image processing method described above.

[0031] The present invention also includes: an instruction acquisition unit that acquires an instruction input by a user; an image search unit that searches the image group for an image that matches a search condition input in response to an instruction from a user, as a search image; an image display unit that displays images included in the image group on a display unit; an image designation unit that designates one image selected in response to an instruction from a user from among the images displayed on the display unit; the instruction acquisition unit, the image search unit, the image display unit, and the image designation unit are hardware or a processor that executes a program; The search conditions are conditions for searching for images from among the image group based on conditions other than the shooting time, The image designation unit designates one search image selected in response to an instruction from the user from among the search images displayed on the display unit as a designated search image; The image display unit, when a specified search image is designated, displays only the specified search image on the display unit, and then, in response to an instruction from a user, displays the specified search image and at least a portion of images included in an image group that were taken before or after the shooting time of the specified search image in chronological order as time-series images on the display unit.

[0032] Here, the image search unit: a tag information display unit that displays tag information assigned to images included in the image group on a display unit; a tag information designation unit that designates tag information selected from the tag information displayed on the display unit in response to an instruction from a user as selected tag information; an image extraction unit that extracts, from a group of images, an image to which the selected tag information has been added as a search image; the tag information display unit, the tag information designation unit, and the image extraction unit are hardware or a processor that executes a program; the tag information display unit displays at least a part of all the tag information assigned to all the images included in the image group on the display unit; the tag information designation unit designates, as a first selected tag information, a first piece of tag information selected in response to an instruction from the user from among the tag information displayed on the display unit; The image extraction unit extracts an image to which the first selection tag information is added as a first search image from the image group; The image display unit displays at least a part of all the first search images on the display unit; It is preferable that the tag information display unit displays at least a part of all the tag information assigned to all the first search images on the display unit. Effect of the Invention

[0033] According to the present invention, the user does not need to return to the image list screen in order to display images taken before and after the shooting time of a search image, but can view images taken before and after the shooting time of a specified search image in chronological order in response to instructions from the user. [Brief description of the drawings]

[0034] [Figure 1] 1 is a block diagram illustrating a configuration of an image processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating a configuration of a client according to an embodiment. [Diagram 3] 11 is a flowchart illustrating an embodiment of the operation of a client. [Figure 4] FIG. 13 is a conceptual diagram illustrating an embodiment of a home screen of an application displayed on a display unit of a client. [Diagram 5] FIG. 13 is a conceptual diagram illustrating a viewing screen for a designated image according to an embodiment. [Figure 6] FIG. 13 is a conceptual diagram illustrating an embodiment of a viewing screen for information on a specified image. [Figure 7] FIG. 13 is a conceptual diagram illustrating an embodiment of an editing screen for information on a specified image. [Figure 8] FIG. 13 is a conceptual diagram of an embodiment showing an input screen for new tag information. [Figure 9] FIG. 13 is a conceptual diagram illustrating a tag information selection screen according to an embodiment. [Figure 10] FIG. 13 is a conceptual diagram of an embodiment showing a state in which first tag information is selected on a tag information selection screen. [Figure 11] 13 is a conceptual diagram of an embodiment showing a state in which the contents of tag information are updated in accordance with first tag information on a tag information selection screen. FIG. [Figure 12] FIG. 11 is a conceptual diagram of an embodiment showing a state in which second tag information is selected on the tag information selection screen. [Figure 13] FIG. 13 is a conceptual diagram illustrating an image list screen according to an embodiment. [Figure 14] FIG. 13 is a conceptual diagram of an embodiment showing a state in which one search image is selected on an image list screen. [Figure 15] FIG. 13 is a conceptual diagram illustrating an embodiment of a viewing screen for a specified search image. [Figure 16] FIG. 13 is a conceptual diagram illustrating an embodiment of a list screen of time-series images. [Figure 17] FIG. 13 is a conceptual diagram of an embodiment illustrating a state in which a "Search Tag" button is tapped on a tag information selection screen. [Figure 18] FIG. 13 is a conceptual diagram illustrating a tag information search screen according to an embodiment. [Figure 19] FIG. 13 is a conceptual diagram illustrating a tag information selection screen according to an embodiment. [Figure 20]FIG. 13 is a conceptual diagram showing an example of a collage image created using several images selected from a group of images dated “2015 / 01 / 10” owned by a user. [Figure 21] This is a conceptual diagram of an example showing a group of images taken on "Saturday, January 10, 2015," the same date as the images used in the collage image. [Figure 22] FIG. 13 is a conceptual diagram of an example showing an image corresponding to the keyword "soccer." DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing system, an image processing method, a program, and a recording medium according to the present invention will be described in detail below based on preferred embodiments shown in the accompanying drawings.

[0036] Fig. 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention. The image processing system 10 shown in Fig. 1 includes a server 12 and a plurality of clients 14 each connected to the server 12 via a network 16.

[0037] The server 12 has functions such as acquiring a group of images owned by each of multiple users who use the image processing system 10, storing each image included in the group of images owned by each user, and sharing the images only with sharing partners designated by each user. The server 12 is not limited to one, but may be multiple, and is configured by a workstation or the like having a control device, a storage device, a communication device, and the like.

[0038] The client 14 is an image processing device of the present invention, and has a function of searching for images from a group of images owned by a user based on search conditions, more specifically, using tag information assigned to the images as image search conditions, and displaying the images resulting from the search on a display unit. The client 14 is configured by a desktop PC (personal computer), a notebook PC, a tablet PC, or a portable terminal such as a mobile phone or a smartphone, which has a control device, an input device, a storage device, a communication device, a display, and the like. The following description will be given taking as an example a case where the client 14 is a smartphone.

[0039] Fig. 2 is a block diagram showing the configuration of a client according to an embodiment. The client 14 shown in Fig. 2 includes an instruction acquisition unit 18, an image storage unit 20, an image analysis unit 22, an evaluation value assignment unit 24, a tag information assignment unit 26, a weighting setting unit 28, a tag information display unit 30, a tag information designation unit 32, an image extraction unit 34, a tag information search unit 36, an image display unit 38, an image designation unit 40, an image information display unit 42, an image information editing unit 44, and a display unit 46.

[0040] In the client 14 , first, the instruction acquisition unit 18 acquires an instruction input by the user of the client 14 .

[0041] A user can input various instructions by touching the touch panel of the smartphone. Note that instructions can also be input using input devices other than the touch panel, such as a mouse, a keyboard, and a track pad. Instructions from the user, which will be described in detail later, include instructions to select tag information, instructions to select an image, instructions to input tag information to be assigned to the image, instructions to input search conditions (keywords) for the tag information, instructions to display the image on display unit 46, and the like.

[0042] The image storage unit 20 is a storage device such as a semiconductor memory, and stores a group of images.

[0043] The image group is taken by a user using the smartphone camera and is stored inside the smartphone. Note that the image group is not particularly limited, and may be, for example, an image group imported into the smartphone from an external source, an online image group stored in a social network system (SNS) or online storage, or a combination of these image groups.

[0044] For example, images taken by a smartphone camera can be set to be automatically uploaded from the smartphone to the server 12. The images uploaded to the server 12 are stored in the storage area of ​​each user in the storage device of the server 12. The images uploaded to the server 12 can be set to be left in the image storage unit 20 of the smartphone, or can be set to be deleted. Also, for example, images taken by a digital camera, images stored in a PC, notebook PC, tablet PC, etc., and images scanned from photo prints, etc. (digital image data) can be imported into the smartphone. Images imported into the smartphone from the outside are stored in the image storage unit 20 and are similarly treated as part of the image group.

[0045] Next, the image analysis unit 22 analyzes each image included in the image group.

[0046] Image analysis items are not particularly limited, but may include, for example, analysis of image content such as detection of subjects appearing in an image (human detection and object detection), detection of a human face, detection of a human expression, detection of a human behavior, detection of a scene (night view, sea, beach, sky, etc.), detection of an event (sports meet, wedding, graduation ceremony, etc.), and detection of user preferences (hobbies), analysis of image quality such as image brightness, color, contrast, and degree of blur, and analysis of time information (time of shooting) and location information (location of shooting) contained in additional information of an image such as Exif (Exchangeable Image File Format).

[0047] Next, the evaluation value assigning unit 24 assigns an evaluation value to each image based on the results of the analysis of each image.

[0048] The method by which the evaluation value assignment unit 24 assigns an evaluation value to an image is not particularly limited, but for example, a score can be assigned to the result of each analysis of the image, a total score of the scores of the results of the multiple analyses can be calculated, and this total score can be assigned as the evaluation value for the image. For example, an image in which a person is smiling will be given a higher score than a crying face, an angry face, etc. Also, an image with appropriate brightness will be given a higher score than an image that is too bright or too dark. In addition, the evaluation value assignment unit 24 sets a higher evaluation value for an image assigned with tag information selected later in time than for an image assigned with tag information selected earlier, depending on the weighting setting of the tag information described below.

[0049] Next, the tag information assigning unit 26 assigns tag information to each image based on the results of the analysis of each image.

[0050] The tag information is various information related to an image, such as the date and place of shooting of the image, the name of the subject appearing in the image, the name of the subject's action, the name expressing the scene, event, person's emotion, etc. read from the image, the name of the user's preference, etc. The tag information is text information, and it is possible to search for the tag information by keyword. The tag information may include tag information of a higher-level concept having tag information of multiple lower-level concepts, for example, tag information of a higher-level concept "sweets" having tag information of two lower-level concepts "Mont Blanc" and "strawberry shortcake". The tag information may also have reliability information indicating the reliability of the tag information. Although the automatically assigned tag information may not be accurate, the accuracy of the tag information can be determined based on the reliability information. The tag information assignment unit 26 is not a required component, but may assign tag information to tag information already assigned to an image, or may assign new tag information to an image to which no tag information has been assigned.

[0051] Next, the weight setting unit 28 sets weights for the tag information.

[0052] Although the method by which the weight setting unit 28 sets weights for tag information is not particularly limited, when a user searches for an image using two or more pieces of tag information, it is considered that tag information selected later in time is more likely to be important than tag information selected earlier in time. In response to this, the weight setting unit 28 can set a heavier weight for tag information selected later in time than for tag information selected earlier in time among two or more pieces of tag information simultaneously selected in response to an instruction from the user from among the tag information displayed on the display unit 46.

[0053] Next, the tag information display unit 30 causes the display unit 46 to display the tag information assigned to the images included in the image group.

[0054] Although there is no particular limitation on the method by which the tag information display unit 30 displays the tag information, for example, the tag information displayed on the display unit 46 can be classified into a plurality of categories and displayed. This allows the tag information to be classified and displayed for each type of tag information, so that the user can easily find tag information that he or she wishes to use as an image search condition from among the tag information displayed on the display unit 46.

[0055] In this case, the tag information display unit 30 can display tag information included in at least one of the multiple categories in descending order of the number of images to which the tag information is attached. This allows tag information attached to many images to be preferentially displayed. Note that tag information represented by consecutive numbers, such as the shooting date of an image, can be displayed in descending or ascending order of the numbers.

[0056] In addition, the tag information display unit 30 may display tag information included in at least one of the multiple categories in descending order of the number of times it has been used as a keyword for searching for images, or, if the tag information has reliability information, may display tag information included in at least one of the multiple categories in descending order of reliability of the tag information based on the reliability information of the tag information.

[0057] In addition, when tag information of a higher concept is selected in response to an instruction from the user, the tag information display unit 30 can display, on the display unit 46, tag information of multiple lower concepts included in the tag information of the higher concept. Furthermore, the tag information display unit 30 can display the tag information as a button (link) on the display unit (touch panel) 46. This allows the user to easily select tag information.

[0058] Next, the tag information designation unit 32 designates the tag information selected from the tag information displayed on the display unit 46 in response to an instruction from the user as selected tag information. Furthermore, the tag information designation unit 32 designates search tag information selected from the search tag information displayed on the display unit 46 in response to an instruction from the user as selected tag information.

[0059] Next, the image extraction unit 34 extracts, from the group of images, images to which the selected tag information designated by the tag information designation unit 32 has been added, as search images.

[0060] Here, the tag information display unit 30, the tag information designation unit 32 and the image extraction unit 34 constitute the image search unit of the present invention, which searches for images that match the search criteria input in response to instructions from the user from among a group of images as search images.

[0061] Next, the tag information search unit 36 ​​searches for tag information that at least partially matches the keyword input in response to instructions from the user from among all tag information assigned to all images included in the image group, as search tag information.

[0062] Next, the image display unit 38 causes the display unit 46 to display the images included in the image group.

[0063] The image display unit 38 displays, for example, each image included in the image group, a list of images included in the image group (thumbnail images), a list of search images (thumbnail images), and the like. When displaying a list of images or a list of searched images on display unit 46, image display unit 38 can display the images in chronological order of their shooting times. Furthermore, when each image included in the image group has information on the image evaluation value, image display unit 38 can also display the images on display unit 46 in descending order of the image evaluation value.

[0064] Next, the image designation unit 40 designates one image selected from the images displayed on the display unit 46 in response to an instruction from the user.

[0065] The user can select one image that he or she wishes to view from the list of images displayed on the display unit 46. When one image is selected, the image display unit 38 displays only this selected one image on the display unit 46.

[0066] Next, the image information display unit 42 causes the display unit 46 to display information on one image designated by the image designation unit 40, that is, one image selected in response to an instruction from the user.

[0067] The image information includes various information related to the image. In the present embodiment, the information includes the time the image was taken and tag information added to the image. However, the information is not limited to this, and may include the image file name, image size, resolution, and information on the location where the image was taken.

[0068] Next, the image information editing unit 44 edits the information of one image displayed on the display unit 46 by the image information display unit 42 in response to an instruction from the user.

[0069] The user can edit the tag information etc. that has been assigned to the image. There is a possibility that the tag information automatically assigned by the tag information assigning unit 26 may include inaccurate tag information. The user can correct or delete tag information that is considered to be inaccurate. The user can also assign new tag information to an image. In this case, the image information editing unit 44 assigns the new tag information input in response to an instruction from the user to a single image displayed on the display unit 46 as information of the single image.

[0070] Next, the display unit 46 displays various information.

[0071] Images, image information, a list of images, a list of tag information, etc. are displayed on the display unit 46. The display unit 46 is a liquid crystal panel, an organic EL (Electro Luminescence) panel, etc., and in this embodiment, is a touch panel.

[0072] Next, with reference to the flowchart shown in FIG. 3, an operation will be described in which the client 14 searches for a desired image from among a group of images using tag information added to the image as a search condition.

[0073] First, the operation when the first tag information is selected will be described. In this case, first, the tag information display section 30 displays at least a portion of all the tag information attached to all the images included in the image group on the display section 46 (step S1). Once the tag information is displayed on the display unit 46, the tag information designation unit 32 then designates the first piece of tag information selected from the tag information displayed on the display unit 46 in response to an instruction from the user as the first selected tag information (selected tag information) (step S2). When the first selection tag information is specified, the image extractor 34 subsequently extracts, from the image group, an image to which the first selection tag information has been added as a first search image (search image) (step S3). After the first search images are extracted, the image display unit 38 subsequently displays at least a portion of all the first search images on the display unit 46 (step S4). Next, at least a part of all the tag information assigned to all the first search images is displayed on the display unit 46 by the tag information display unit 30 (step S5).

[0074] Next, an operation will be described in a case where second tag information is selected at the same time as the first tag information. In this case, the tag information designation unit 32 then simultaneously designates, in addition to the first selected tag information, a second piece of tag information selected from the tag information displayed on the display unit 46 in response to an instruction from the user as second selected tag information (selected tag information) (step S6). Next, the image extracting unit 34 extracts the first search image to which the second selection tag information has been added as a second search image (search image) from the first search images (step S7). Next, the image display unit 38 displays at least a portion of all the second search images on the display unit 46 (step S8). Next, the tag information display unit 30 displays at least a part of all the tag information assigned to all the second search images on the display unit 46 (step S9).

[0075] The same operation is performed when three or more pieces of tag information are selected at the same time. That is, the operation when two or more pieces of tag information are selected at the same time is as follows. First, the tag information designation unit 32 simultaneously designates two or more pieces of tag information selected from the tag information displayed on the display unit 46 in response to an instruction from the user as selected tag information. Next, the image extracting unit 34 extracts, from the group of images, images to which all of the two or more pieces of selected tag information have been added, as search images. The image display unit 38 then causes at least a portion of all of the retrieved images to be displayed on the display unit 46 . Next, the tag information display unit 30 displays on the display unit 46 at least a portion of all the tag information assigned to all the search images.

[0076] The image processing device of the present invention will be described below by taking as an example a case in which the client 14 is realized by an application program (hereinafter, also simply referred to as an app) that runs on a smartphone.

[0077] Fig. 4 is a conceptual diagram of an embodiment showing a home screen of an application displayed on a display unit of a client. The display unit 46 of the smartphone shown in Fig. 4 is configured with a touch panel, and the user can use the display unit (touch panel) 46 not only to display various information but also as an instruction acquisition unit 18 to acquire various instructions input by the user by touching the touch panel.

[0078] A notification and status area 50 is displayed at the top of the home screen, and a menu area 52 is displayed below that. In addition, a navigation area 54 is displayed at the bottom of the home screen.

[0079] In the navigation area 54, the buttons "Search," "Organize," "Home," "Share," and "Miscellaneous" are displayed in this order from left to right. When the “Search” button is tapped (pressed), a tag information selection screen is displayed on the display unit 46. Furthermore, when the "Combine" button is tapped, a composite image such as a photo album is automatically created using the images included in the image group. When the "Home" button is tapped, a home screen, that is, a list of images included in the image group, is displayed on the display unit 46. In the list of images, the images are displayed, for example, arranged in order of the shooting time (descending or ascending order). When the "Share" button is tapped, the image selected in accordance with the user's instructions is shared with other users. When the "Various" button is tapped, various other functions not displayed in the navigation area can be selected and executed.

[0080] A list of images is displayed in the area between the menu area 52 and the navigation area 54. Before searching for images, in a state in which no tag information has been selected as a search condition, the image list screen displays a list of images, for example at least a portion of all images included in an image group, in chronological order from the top to the bottom of the image list screen in the order of the image shooting time, which in this embodiment is in descending order of the shooting time.

[0081] As shown by the finger mark 56 in FIG. 4, the user can tap an image that he or she wishes to view from among the images displayed on the image list screen, thereby displaying only the tapped image on the display unit 46 and viewing it.

[0082] In this case, the image designation unit 40 designates one image selected from the list of images displayed on the display unit 46 in response to instructions from the user as the designated image, and the image display unit 38 displays only the designated image on the display unit 46.

[0083] Fig. 5 is a conceptual diagram of an embodiment showing the designated image viewing screen. The designated image viewing screen is displayed on the entire surface of the display unit 46. As shown in Fig. 5, an "x" button 58 is displayed in the upper left corner of the designated image viewing screen, and an "information" button 60 is displayed in the lower right corner. A "trash" button 62, a "share" button 64, and the like are displayed in the lower part of the designated image viewing screen.

[0084] For example, the user can close the designated image viewing screen and return to the image list screen (home screen) by tapping the "x" button 58. The user can also delete the designated image by tapping the "trash" button 62, or share the designated image with other users by tapping the "share" button 64. Furthermore, the user can flip the designated image over and view information about the designated image by tapping the "information" button 60, as shown by the fingerprint 56 in FIG. 5.

[0085] 6 is a conceptual diagram of an embodiment showing a viewing screen for designated image information. An "x" button 66 is displayed in the upper left corner of the viewing screen for designated image information, and an "Edit" button 68 is displayed in the upper right corner. A thumbnail image of the designated image is displayed in the upper part of the viewing screen for designated image information, and below that, the shooting time of the designated image, automatically added tags, tags added by the user, and an "Add tag" button 70 are displayed in this order.

[0086] Automatically added tags are categorized into the following categories: calendar, people, location, auto-added tags, and service tags, while tags you add yourself are categorized into the manual tags category. The calendar category includes buttons for tag information such as "2018" and "December" as the year and month the specified image was taken. The People category includes a button for tag information for "Taro." The location category includes buttons for tag information of "Park," "Minato Ward," and "Odaiba Marine Park." The category of automatically assigned tags includes buttons for tag information of "outdoors", "boy" and "sunny". The Scan Tags category includes a button for tag information for "Scan Services." The manually added tag category includes a button for tag information for "kids' outfits."

[0087] For example, the user can close the designated image information viewing screen and return to the designated image viewing screen by tapping the "x" button 66. Also, the user can edit the designated image information by tapping the "Edit" button 68, as indicated by the fingerprint 56.

[0088] Fig. 7 is a conceptual diagram of an embodiment showing an editing screen for the information of a specified image. A "Cancel" button 72 is displayed in the upper left part of the editing screen for the information of a specified image shown in Fig. 7, and a "Save" button 74 is displayed in the upper right part.

[0089] When the "Edit" button 68 is tapped on the viewing screen for the information of the specified image, an "X" button is displayed within the button for tag information that can be deleted on the editing screen for the information of the specified image. When the user taps the "X" button within a tag information button, the button for that tag information is deleted by the image information editing unit 44. Also, as shown by the finger print 56 in Fig. 7, when the user taps the "Add tag" button 70, an input screen for new tag information is displayed.

[0090] When the "Cancel" button 72 is tapped on the editing screen for the designated image information, the edited contents for the designated image information are discarded and the screen returns to the viewing screen for the designated image information. When the "Save" button 74 is tapped, the edited contents for the designated image information are saved.

[0091] Fig. 8 is a conceptual diagram of an embodiment showing an input screen for new tag information. A thumbnail image of a specified image is displayed at the top of the input screen for new tag information shown in Fig. 8, and an input field 76 for new tag information is displayed below it. A message "Add tag" is displayed in the input field 76 for new tag information to prompt the input of new tag information. A "Cancel" button 78 is displayed to the right of the input field 76 for new tag information.

[0092] When the user taps the new tag information input field 76, a history of recently added new tag information is displayed below the new tag information input field 76, and a software keyboard is displayed at the bottom of the new tag information input screen. The user can input the name of the new tag information in the new tag information input field 76 using the software keyboard.

[0093] In this case, each time the user inputs characters in the new tag information input field 76, a partial match search is performed, and tag information that partially matches the characters input in the new tag information input field 76 is searched for among all tag information assigned to all images included in the image group, and displayed in the new tag information input screen. The user can also select new tag information from the list of tag information displayed in the new tag information input screen.

[0094] When the user taps the "Cancel" button 78, the input screen for new tag information is closed, and the screen returns to the edit screen for the information of the specified image.

[0095] Next, the tag information selection screen will be described. As described above, when the “Search” button is tapped, a tag information selection screen is displayed on the display unit 46 .

[0096] 9 is a conceptual diagram of an embodiment showing a tag information selection screen. The tag information selection screen is displayed on top of an area other than a part of the upper part of the image list screen so that the image list in the lower layer can be seen through at a certain transparency. In a state where no tag information is selected, the tag information selection screen displays a part of all tag information assigned to all images included in the image group, classified into multiple categories. Each piece of tag information is displayed as a button in the tag information selection screen. In addition, a "Search for tags" button 80 is displayed at the bottom of the tag information selection screen.

[0097] The multiple categories include calendar, people, photo locations, automatically added tags, manually added tags, and service tags. The Calendar category includes buttons for tag information of the year and month an image was taken. The People category includes buttons for tag information of the names of people appearing in the image. The Location category includes a button for tag information about the location where the image was taken. The automatically assigned tag category includes buttons of tag information that is automatically assigned by the tag information assigning unit 26. The manually added tag category includes buttons of tag information that has been manually added by the user, that is, buttons of tag information that has been added by the image information editing unit 44. The service tag category includes buttons for tag information related to services provided by the image processing system 10 of the present invention, such as scanning a print to create digital image data.

[0098] On the tag information selection screen, in the calendar category, tag information buttons for "2018," "2017," and "2016" are displayed in that order from left to right as the year the image was taken, and tag information buttons for "December," "November," "October," and "September" are displayed in that order from left to right as the month the image was taken. In the person category, buttons for tag information of person names "Takashi," "Akira," and "Fuji Hanako" are displayed in that order from left to right. The Location category does not include tag information. In the automatically assigned tag category, buttons for tag information "bicycle," "ramen," and "drinking party" are displayed in that order from left to right. In the manually added tag category, buttons for tag information such as "Circle," "Today's Lunch," and "Breathtaking View" are displayed in that order from left to right. The Service Tags category displays a button for tag information for "Scanned Images."

[0099] The tag information buttons in each category other than the calendar category are displayed in a horizontal row. In the calendar category, the tag information buttons for the year and month of photographing of the images are displayed in a horizontal row in descending order of the photographing year and month. The tag information buttons in each of the other categories are displayed in descending order of the number of images to which each tag information is applied.

[0100] In addition, if each category contains so many pieces of tag information that it is not possible to display all of them horizontally, the user can tap the button of the tag information included in each category and then swipe left or right to push the button of the tag information displayed on the tag information selection screen outward and move the button of the tag information that is not displayed on the tag information selection screen into the tag information selection screen and display it.

[0101] The user does not need to type in keywords to search for images; instead, the user can tap and select tag information corresponding to the desired image to be searched for from the tag information displayed on the display unit 46, allowing for intuitive searches.

[0102] Here, assume that, without any tag information being selected as a search condition, the user taps the tag information button for “bicycle” as the first piece of tag information from among the tag information buttons displayed on the tag information selection screen, as shown by finger print 56 in FIG. 10.

[0103] In this case, the tag information of "Bicycle" corresponding to the button of the tag information of "Bicycle" selected in response to an instruction from the user is designated as the first selected tag information by the tag information designation unit 32. Also, the tag information display unit 30 changes the color of the button of the tag information of "Bicycle", and displays "Bicycle" on the left side in the upper area of ​​the list screen of images that is not covered by the tag information selection screen.

[0104] Next, the image extraction unit 34 extracts images from the group of images that have the tag information of "bicycle", which is the first selected tag information, as first search images, and the image display unit 38 displays at least a portion of all of the first search images on an image list screen.

[0105] 11, at least a part of all the tag information assigned to all the first search images is displayed on the tag information selection screen by the tag information display unit 30. That is, the content of the tag information displayed on the tag information selection screen is updated.

[0106] In this embodiment, the calendar category is changed from "2018", "2017" and "2016" to buttons with tag information of "2018" and "2017" as the years in which the images were taken. Also, the person category is changed from "Takashi," "Akira," and "Fuji Hanako" to buttons for tag information of "Takashi," "Akira," and "Yamada-san." The categories of automatically assigned tags will be changed from "bicycle," "ramen," and "drinking party" to buttons with tag information for "bicycle," "mountain," and "guardrail." The categories of manually added tags will be changed from "Circle," "Today's Lunch," and "Amazing View" to buttons with tag information for "Circle," "Amazing View," and "Bicycle Trip."

[0107] In other words, when only the "bicycle" tag information button is selected from among the tag information buttons displayed on the tag information selection screen, the image group is narrowed down to only search images that have the tag information of "bicycle" assigned to them, and, from all tag information assigned to all images included in the image group, the search results are narrowed down to only all tag information assigned to all search images that have the tag information of "bicycle" assigned to them.

[0108] 12, suppose that the user taps the button for tag information "Bicycle Trip" as the second piece of tag information in addition to the button for the first tag information "Bicycle" from among the tag information buttons displayed on the tag information selection screen. In other words, suppose that with the button for tag information "Bicycle" selected, the button for tag information "Bicycle Trip" is additionally selected, and two pieces of tag information are selected at the same time.

[0109] In this case, in addition to the first tag information "Bicycle", the tag information designation unit 32 designates the second tag information "Bicycle Trip" corresponding to the button of the tag information "Bicycle Trip" selected in response to an instruction from the user as the second selected tag information. In other words, both the tag information "Bicycle" and the tag information "Bicycle Trip" are simultaneously designated as selected tag information. Also, the tag information display unit 30 changes the color of the button of the tag information "Bicycle Trip" in addition to the button of the tag information "Bicycle", and displays "Bicycle Trip" to the right of "Bicycle" displayed in the upper area of ​​the list screen of images that is not covered by the tag information selection screen.

[0110] Next, the image extraction unit 34 extracts the first search image that has been assigned the tag information of "Bicycle Trip" from among the first search images that have been assigned the tag information of "Bicycle" as the second search image, and the image display unit 38 displays at least a portion of the second search image on the image list screen, as shown in FIG. 13. Although not shown, the tag information display unit 30 displays at least a part of all the tag information assigned to all the second search images on a tag information selection screen.

[0111] In other words, when the buttons for both tag information "bicycle" and "bicycle trip" are selected simultaneously, the image group is narrowed down to only search images that have both tag information "bicycle" and "bicycle trip" assigned to them, and, among all tag information assigned to all images included in the image group, the search is narrowed down to only all tag information assigned to all search images that have both tag information "bicycle" and "bicycle trip" assigned to them.

[0112] The user can close the tag information selection screen at any time by swiping downward on the downward arrow 92 displayed at the top of the tag information selection screen. When the tag information selection screen is closed, a list screen of images in the area covered by the tag information selection screen is displayed, as shown in Fig. 13. This allows the user to view a list of search images.

[0113] When the image list screen contains more search images than can be displayed all at once, the user can swipe the search image displayed on the image list screen up or down to move the search image up or down, thereby pushing a portion of the search image displayed on the image list screen outward and moving a portion of the search image that is not displayed on the image list screen into the image list screen for display.

[0114] Furthermore, when the tag information selection screen is closed, an upward arrow 94 is displayed at the top of the tag information selection screen as shown in Fig. 13. The user can open the tag information selection screen at any time by swiping this upward arrow 94 upward. In this embodiment, the tag information selection screen can be switched between three stages: a fully open state shown in Figs. 9 to 12, a fully closed state shown in Fig. 13, and an intermediate position between the two. The tag information selection screen may be switched between three or more stages.

[0115] In addition, the user can tap the "Search" button at any time to display the tag information selection screen when the image list screen is displayed. Alternatively, when the two tag information items "Bicycle" and "Bicycle Trip" are selected, the user can select additional tag information buttons to further narrow down the tag information and thereby further narrow down the images.

[0116] The two tag information items, "bicycle" and "bicycle trip", designated as selected tag information are maintained as selected tag information until the tag information designation unit 32 cancels the designation as selected tag information in response to instructions from the user. For example, a "Cancel" button may be provided on the tag information selection screen, and the user may tap the "Cancel" button to simultaneously cancel the selection of the two tag information items, "Bicycle" and "Bicycle Trip", that have been specified as selected tag information.

[0117] Furthermore, each time the user taps a tag information button, the user can switch between selecting and canceling the tag information as selected tag information. In this case, the tag information designation unit 32, in response to instructions from the user, switches between designating the tag information corresponding to the button as selected tag information and canceling the designation of the tag information corresponding to the button as selected tag information each time the button is pressed.

[0118] In other words, the selection of tag information is maintained until it is released, so that while the selection of tag information is maintained, that is, after an image search is performed using the selected tag information, if the display screen is switched to another screen and then switched back to the tag information selection screen, the search image that is the result of the search using the selected tag information can be viewed at any time.

[0119] In addition, when the tag information is displayed as buttons on the display unit 46, the user can press and hold (long tap) a button of one piece of tag information to select tag information that is a superordinate concept of the tag information of the button that was pressed and held.

[0120] In this case, when a button of tag information selected in response to a user's instruction from among the buttons of tag information displayed on the display unit 46 is pressed for a certain period of time or more, the tag information designation unit 32 designates all tag information of subordinate concepts contained in the tag information of the superordinate concept of the tag information corresponding to the pressed button of the tag information as selected tag information. For example, when the button for tag information of "Mont Blanc" is pressed and held, the tag information of all "Sweets", which is a superordinate concept of "Mont Blanc", is selected.

[0121] Alternatively, when the tag information designation unit 32 selects one piece of tag information from among the tag information displayed on the display unit 46 in response to an instruction from the user, the designation of other tag information that has already been designated may be cancelled, and only the selected one piece of tag information may be designated as the selected tag information.

[0122] Next, the user can tap on one search image that the user wants to view from among the search images displayed on the image list screen, as shown by finger mark 56 in FIG. 14, to display only the tapped search image on display unit 46 for viewing.

[0123] In this case, the image designation unit 40 designates one search image selected from the search images displayed on the display unit 46 in response to instructions from the user as the designated search image, and the image display unit 38 displays only the designated search image on the display unit 46.

[0124] Fig. 15 is a conceptual diagram of an embodiment showing a viewing screen for a specified search image. The viewing screen for a specified search image is displayed on the entire surface of the display unit 46. An "x" button 58 is displayed in the upper left corner of the viewing screen for a specified search image shown in Fig. 15, a "Calendar" button 82 is displayed in the lower left corner, and an "Information" button 60 is displayed in the lower right corner. A "Trash" button 62, a "Share" button 64, and an "Other" button 84 are displayed in the lower part of the viewing screen for a specified search image. The designated search image viewing screen has the same configuration as the designated image viewing screen, except that a "Calendar" button 82 is displayed in the lower left portion.

[0125] For example, the user can close the designated search image viewing screen and return to the search image list screen by tapping the "x" button 58. The user can also tap the "trash" button 62 to delete the designated search image, the "share" button 64 to share the designated search image with other users, or the "other" button 84 to perform other functions. The user can also view information about the designated search image by tapping the "information" button 60.

[0126] Furthermore, the user can view the searched images by displaying them one by one in sequence. In this case, after only the specified search image is displayed on the display unit 46, each time an instruction is input from the user, the image display unit 38 sequentially displays on the display unit 46, from among all the search images, only one search image that was taken before or after the shooting time of the specified search image, in the order of the search images taken in chronological order from the shooting time of the specified search image.

[0127] For example, when a user swipes a designated search image to the right, only one search image captured at an earlier capture time than the designated search image is displayed from among all search images. After this, each time the search image is swiped to the right in response to a user's instruction, only the previous search image is displayed in order of the search images captured in chronological order from the capture time of the designated search image.

[0128] On the other hand, when the user swipes the designated search image to the left, only one search image captured at a shooting time one after the designated search image is displayed from among all the search images. After this, every time the search image is swiped to the left in response to a user's instruction, only one search image is displayed in sequence, one after the other, in the order of the search images captured in chronological order from the shooting time of the designated search image.

[0129] In addition, as shown in FIG. 15, the user can tap the “Calendar” button 82 displayed on the search image viewing screen to display on the display unit 46 images taken before and after the shooting time of the specified search image from the image group for viewing. In this case, when a specified search image is designated, only the specified search image is displayed on the display unit 46, and then, in response to instructions from the user, the image display unit 38 displays the specified search image and at least some of the images included in the image group that were taken before or after the shooting time of the specified search image in chronological order on the display unit 46 as time-series images.

[0130] In this way, the user does not need to return to an image list screen such as a home screen in order to display images taken before and after the shooting time of a search image. In response to instructions from the user, in this embodiment, with a single action of tapping the "Calendar" button 82, the user can view images taken before and after the shooting time of a specified search image in chronological order.

[0131] Here, the same thing can be achieved by returning to an image list screen such as a home screen in response to a user's instruction and displaying images taken before and after the specified search image in chronological order. However, with this type of operation, if a large number of images spanning several years are stored in the smartphone, the user is forced to move between images spanning several years on an image list screen such as a home screen until the date of capture of the desired image.

[0132] In contrast, by using the image processing device of this embodiment, it is possible to display and view images taken before and after the shooting time of the specified search image in chronological order with a single action of tapping the "Calendar" button 82, which is extremely convenient compared to the case where, using conventional technology, as described above, images taken before and after the shooting time of the specified search image are displayed in chronological order by returning to an image list screen such as a home screen.

[0133] 16 is a conceptual diagram showing an embodiment of a list screen of time-series images. The list screen of time-series images is displayed in the area between the menu area 52 and the navigation area .

[0134] Similarly, the user can tap a time-series image that he or she wishes to view from among the time-series images displayed on the list screen of time-series images, thereby displaying only the tapped time-series image on the display unit 46 and viewing it. In this case, the image designation unit 40 designates one time-series image selected from the time-series images displayed on the display unit 46 in response to instructions from the user as the designated time-series image, and the image display unit 38 displays only the designated time-series image on the display unit 46.

[0135] Furthermore, when a specified time-series image is designated, only the specified time-series image is displayed on the display unit 46, and then, each time an instruction is input from the user by the image display unit 38, only one image from the group of images that was taken before or after the shooting time of the specified time-series image is displayed on the display unit 46 in the order of the images taken in chronological order from the shooting time of the specified time-series image. The operation when the designated time-series image is swiped to the right or left is similar to the operation when the designated search image is swiped to the right or left.

[0136] Instead of providing the "Calendar" button 82, a specific touch operation may be provided to achieve the same function as the "Calendar" button 82.

[0137] The user can return to the search image list screen by tapping the "Home" button on the time-series image list screen. As mentioned above, the selection of tag information is maintained until it is deselected, so when you return to the search image list screen, a list of search images that are the results of a search using the selected tag information is displayed on the search image list screen.

[0138] In addition, before displaying the time-series images on the display unit 46, the search conditions for searching for images from the image group are not particularly limited as long as they are conditions for searching for images from the image group based on conditions other than the image shooting time. The conditions for the image shooting time include not only the image shooting time but also various conditions related to the image shooting time, such as the year, month, week, date and time the image was shot, and the like, and the conditions other than the image shooting time include all conditions other than the conditions related to the image shooting time. As search conditions, the analysis results of the image by the image analysis unit 22 and the names of the tag information assigned by the tag information assignment unit 26 can be used, and examples include the names of the subjects appearing in the image (person's name and object name), the names of the person's facial expressions, the names of the person's actions, the names of the person's emotions, the names of scenes, the names of events, the names of the user's preferences, the names of the shooting locations, etc.

[0139] In addition, if the user cannot see the button for the tag information that the user wants to use as a search condition on the tag information selection screen, the user can tap the “Search Tags” button 80, as shown by the finger print 56 in FIG. 17, to enter keywords and search for tag information.

[0140] When the "Search Tag" button 80 is tapped, a tag information search screen is displayed on the display unit 46, as shown in FIG.

[0141] The tag information search screen is displayed on top of the image list screen except for a portion of the upper area, so that the list of searched images in the lower layer is visible through it at a certain transparency. A keyword input field 86 is displayed at the top of the tag information search screen. A message "Search tags" is displayed in the keyword input field 86 to prompt the user to enter a keyword. An "x" button 88 is displayed at the right end of the keyword input field 86, and a "cancel" button 90 is displayed to the right of the keyword input field 86.

[0142] Next, it is assumed that the user inputs a keyword in the keyword input field 86 .

[0143] In this case, the tag information search unit 36 ​​searches for tag information that at least partially matches the keyword input in response to a user's instruction from among all tag information assigned to all images included in the image group, as search tag information. Next, the tag information display section 30 displays at least a part of the search tag information on the display section 46. Next, the tag information designation unit 32 designates one piece of search tag information selected from the search tag information displayed on the display unit 46 in response to an instruction from the user. Next, the image extracting unit 34 extracts an image to which one piece of search tag information has been added as a search image from the group of images. The image display unit 38 then causes at least a portion of all of the retrieved images to be displayed on the display unit 46 . Next, the tag information display unit 30 displays on the display unit 46 at least a portion of all the tag information assigned to all the search images.

[0144] For example, as shown in FIG. 18, when "a" is entered into keyword input field 86 to search for tag information for "ice cream," a partial match search is performed, and tag information that partially matches "a" is searched for as search tag information from among all tag information assigned to all images included in the image group.

[0145] Next, as candidates for search tag information, as shown in FIG. 18, on the tag information search screen, buttons for tag information such as "Ai-chan," "A-chan," "Amaou," and "Ice Cream" are displayed in that order from top to bottom.

[0146] When the tag information search screen contains more search tag information than can be displayed all at once, the user can swipe the search tag information displayed on the tag information search screen up or down to move the list of candidate search tag information up or down, thereby pushing outward some of the search tag information displayed on the tag information search screen and moving inward and displaying some of the search tag information that is not displayed on the tag information search screen.

[0147] Furthermore, the user can delete all the keywords entered in the input field 86 at once by tapping an "x" button 88. Furthermore, the user can close the tag information search screen and return to the tag information selection screen by tapping a "cancel" button 90. Furthermore, the user can close the tag information search screen at any time by swiping the downward arrow 96 displayed at the top of the tag information search screen.

[0148] Next, suppose that the user taps, for example, the button for tag information "ice cream" from among the buttons of candidate search tag information displayed on the tag information search screen, as indicated by a finger print 56 in FIG.

[0149] In this case, as shown in Fig. 19, a tag information selection screen is displayed on the display unit 46. The subsequent operations are the same as those in the case where no tag information is selected and the button for tag information of "Bicycle" is selected.

[0150] That is, the tag information of "Ice Cream" is specified as the selected tag information, and "Ice Cream" is displayed on the left side of the upper area of ​​the list screen of images that is not covered by the tag information selection screen, as shown in FIG. Next, images to which the tag information of "ice cream", which is the selected tag information, is added are extracted as search images from the image group, and the search images are displayed on a list screen of images. Furthermore, all tag information attached to the search image to which the tag information "ice cream" that is the selected tag information has been attached is displayed on the tag information selection screen.

[0151] The operation from this point onwards is the same as when the button for tag information "Bicycle" is selected and then the button for tag information "Bicycle Trip" is additionally tapped, allowing the user to select additional buttons for tag information that they wish to use as search conditions.

[0152] In addition, when a user takes a new image, adds new tag information to an image, or deletes tag information already added to an image, the reliability of the tag information may be improved based on these new images, new tag information, deleted tag information, etc., through machine learning using, for example, AI (Artificial Intelligence).

[0153] In addition, the server 12 may be provided with at least some of the functions provided by the client 14, such as image analysis, assignment of evaluation values, assignment of tag information, setting weighting, image extraction, and search for tag information, and the server 12 may execute at least some of the functions.

[0154] In the device of the present invention, the hardware configuration of the processing unit that executes various processes, such as the instruction acquisition unit 18, image analysis unit 22, evaluation value assignment unit 24, tag information assignment unit 26, weighting setting unit 28, tag information display unit 30, tag information designation unit 32, image extraction unit 34, tag information search unit 36, image display unit 38, image designation unit 40, image information display unit 42 and image information editing unit 44, may be dedicated hardware or various processors or computers that execute programs.

[0155] Various types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically for performing specific processing.

[0156] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Also, multiple processing units may be configured with one of the various processors, or two or more of the multiple processing units may be combined into one processor.

[0157] For example, as typified by computers such as servers and clients, one processor is configured by combining one or more CPUs and software, and this processor functions as multiple processing units. Also, as typified by systems on chips (SoCs), there is a form in which a processor is used to realize the functions of the entire system including multiple processing units in a single IC (Integrated Circuit) chip.

[0158] Furthermore, the hardware configuration of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0159] Furthermore, the method of the present invention can be implemented, for example, by a program for causing a computer to execute each of the steps. Also, a computer-readable recording medium on which this program is recorded can be provided.

[0160] Although the present invention has been described in detail above, the present invention is not limited to the above-described embodiment, and various improvements and modifications may be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0161] 10 Image Processing System 12 Server 14 Client 16 Network 18 Instruction acquisition part 20 Image storage unit 22 Image Analysis Section 24 Evaluation value assignment section 26 Tag information assignment section 28 Weighting setting section 30 Tag information display section 32 Tag information specification section 34 Image Extraction Section 36 Tag information search section 38 Image display section 40 Image specification section 42 Image information display section 44 Image Information Editorial Department 46 Display section 50 Notification and Status Area 52 Menu Area 54 Navigation Area 56 Fingerprint 58, 66, 88 "X" buttons 60 "Information" button 62 "Trash" button 64 "Share" button 68 "Edit" button 70 "Add tag" button 72, 78, 90 "Cancel" button 74 "Save" button 76 New tag information input field 80 "Search Tags" button 82 "Calendar" button 84 "Other" button 86 Keyword input field 92, 94, 96 Arrows

Claims

1. An instruction acquisition unit that acquires an instruction input by a user; an image search unit that searches for an image matching a search condition inputted in response to an instruction from the user from among the group of images as a search image; an image display unit that displays images included in the image group on a display unit, the search conditions are conditions for searching for images from the group of images based on conditions other than the shooting time, the image display unit displays only the search image on the display unit, and then, in response to an instruction from the user with a single action, displays the search image and at least a portion of images included in the image group that were taken on shooting dates before and after the shooting date of the search image on the display unit in chronological order as time-series images; The image processing system wherein the user's single action instruction is input by pressing a button displayed on a viewing screen for the search image.

2. The image search unit includes: a tag information display unit that displays tag information assigned to the images included in the image group on the display unit; a tag information designation unit that designates tag information selected from the tag information displayed on the display unit in response to an instruction from the user as selected tag information; an image extraction unit that extracts, from the group of images, an image to which the selected tag information has been added as the search image; the tag information display unit displays, on the display unit, at least a portion of all of the tag information assigned to all of the images included in the image group; the tag information designation unit designates, as first selected tag information, a first piece of tag information selected in response to an instruction from the user from among the tag information displayed on the display unit; The image extraction unit extracts an image to which the first selection tag information is added as a first search image from the group of images, The image display unit displays at least a portion of all of the first search images on the display unit, The image processing system according to claim 1 , wherein the tag information display unit causes at least a part of all the tag information assigned to all the first search images to be displayed on the display unit.

3. the tag information designation unit simultaneously designates, in addition to the first selected tag information, second tag information selected from the tag information displayed on the display unit in response to an instruction from the user as second selected tag information; the image extraction unit extracts, from the first search images, the first search image to which the second selection tag information has been added as a second search image; The image display unit displays at least a portion of all of the second search images on the display unit, The image processing system according to claim 2 , wherein the tag information display unit causes at least a part of all the tag information assigned to all the second search images to be displayed on the display unit.

4. the tag information designation unit simultaneously designates, as the selected tag information, two or more pieces of tag information selected in response to an instruction from the user from among the tag information displayed on the display unit; the image extraction unit extracts, from the group of images, an image to which all of the two or more pieces of selected tag information have been added as the search image; The image display unit displays at least a portion of all of the search images on the display unit, 4. The image processing system according to claim 2, wherein the tag information display unit displays at least a part of all the tag information assigned to all the searched images on the display unit.

5. An instruction acquisition unit acquires an instruction inputted by a user; an image search unit searches for an image matching the search criteria input in response to an instruction from the user from among the images as a search image; and causing an image display unit to display images included in the image group on a display unit, the search conditions are conditions for searching for images from the group of images based on conditions other than the shooting time, After displaying only the search image on the display unit, in response to an instruction from the user with a single action, displaying the search image and at least a portion of images included in the image group that were taken on shooting dates before and after the shooting date of the search image in chronological order as time-series images on the display unit; The image processing method, wherein the instruction from the user in one action is input by pressing a button displayed on a viewing screen for the search image.

6. 6. A program for causing a computer to execute each step of the image processing method according to claim 5.

7. 6. A computer-readable recording medium having recorded thereon a program for causing a computer to execute each step of the image processing method according to claim 5.

Images