Image processing device, image processing device operating method, and image processing device operating program

Abstract

This image processing device includes a processor which: acquires a captured image obtained by imaging a marker image that is associated with content and that serves as a trigger for outputting the content; and performs image quality correction processing on at least one of the marker image and the captured image to reduce a difference in image quality between the marker image and the captured image.

Description

Image processing device, operation method of image processing device, and operation program of image processing device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for an image processing device, and an operating program for an image processing device.

[0002] The content management system disclosed in Japanese Patent Laid-Open No. 2016-058024 manages managed content consisting of managed images and video data linked to the managed images, and includes a still image data extraction unit, a person of interest detection unit, a movement trajectory detection unit, a motion analysis unit, a still image data output unit, a managed marker storage unit, and a managed image generation unit. The still image data extraction unit extracts multiple frames of still image data from the video data. The person of interest detection unit detects a person of interest, who is a target of processing, from each of multiple still images corresponding to the multiple frames of still image data. The movement trajectory detection unit tracks the movement of the person of interest in the video corresponding to the video data based on the detection result of the person of interest in the multiple still images, thereby detecting a movement trajectory of the person of interest. The movement analysis unit analyzes the movement of the person of interest in the video based on the movement trajectory of the person of interest, and calculates an evaluation value for the movement of the person of interest for each of the multiple still images based on the analyzed movement of the person of interest. The still image data output unit outputs, as output still image data, one still image from among the still image data of multiple frames for which the evaluation value for the movement of the person of interest is equal to or greater than a threshold. The management marker storage unit stores the output still image data or the image feature amount of the output still image corresponding to the output still image data as a management marker, linking it to the moving image data of the moving image corresponding to the output still image. The management image generation unit generates management image data including the output still image data.

[0003] The content playback system described in JP 2016-001821 A includes a server and a mobile terminal connected to the server via a network. The server includes a storage unit that stores, for one piece of video data or each of two or more pieces of video data, selected image data, which is still image data selected from one or more still image data extracted from the video data, or specific information about the selected image data, in association with the video data as a management marker for the selected image corresponding to the selected image data. The mobile terminal includes an image capture unit that captures an output image in which the selected image is output to obtain captured image data. The server also includes an image analysis unit that performs image analysis of the captured image data to obtain a management marker for the captured image corresponding to the captured image data, a management marker identification unit that identifies, from the management markers for the selected images stored in the storage unit, a management marker for the selected image corresponding to the management marker for the captured image as a specific management marker, and a digest video generation unit that extracts a portion of related video data, which is video data associated with the specific management marker, to generate digest video data. The mobile terminal further includes a display unit that plays and displays a digest video corresponding to the digest video data, and a control unit that controls the display unit to play and display a digest video corresponding to the digest video data generated based on the captured image data when an output image is captured by the image capturing unit.

[0004] One embodiment of the technology disclosed herein provides an image processing device, an operating method for an image processing device, and an operating program for an image processing device that can reduce the risk of erroneous authentication caused by the difference in image quality between a marker image that starts content playback when photographed and the captured image of the marker image.

[0005] The image processing device of the present disclosure includes a processor, which acquires a captured image of a marker image associated with content that triggers the output of the content, and performs image quality correction processing on at least one of the marker image or the captured image to reduce the difference in image quality between the marker image and the captured image.

[0006] It is preferable that the processor searches for a marker image corresponding to the captured image that has undergone image quality correction processing, a marker image corresponding to the captured image that has undergone image quality correction processing, or a marker image corresponding to the captured image that has undergone image quality correction processing.

[0007] The processor preferably outputs content associated with the located marker image.

[0008] The captured image is a print image of a print of a marker image, and the processor preferably performs at least one of the following image quality correction processes: processing according to the type of printing medium used to print the marker image; processing according to fading of the marker image print over time; removing the effects of ambient light when the marker image print is photographed; and removing and interpolating noise present in the marker image print that interferes with searching for a marker image corresponding to the captured image.

[0009] The processor preferably derives the similarity between the marker image to be registered and the already registered marker image, and if there is a registered marker image whose similarity with the marker image to be registered is within a set range, either not register the marker image to be registered, or register the marker image to be registered with an identifier to distinguish it from the already registered marker image.

[0010] The processor preferably receives an instruction from the user as to whether or not to assign an identifier to the marker image to be registered and register it.

[0011] The processor preferably associates and registers the content and the marker image.

[0012] The photographed image is a photographed print of a marker image, and the print medium used to print the marker image is preferably instant film.

[0013] The method of operating the image processing device of the present disclosure includes acquiring a captured image of a marker image associated with content that triggers the output of the content, and performing image quality correction processing on at least one of the marker image or the captured image to reduce the difference in image quality between the marker image and the captured image.

[0014] The operating program of the image processing device disclosed herein causes a computer to perform processing including acquiring a captured image of a marker image associated with content that triggers the output of the content, and performing image quality correction processing on at least one of the marker image or the captured image to reduce the difference in image quality between the marker image and the captured image.

[0015] 1 is a diagram illustrating a user terminal, an instant camera, and a video management server. FIG. 1 is a flowchart illustrating the process from shooting a video on a user terminal to playing and displaying the video on the user terminal. FIG. 2 is a block diagram illustrating computers constituting the user terminal and the video management server. FIG. 3 is a block diagram illustrating a processing unit of a CPU of the user terminal. FIG. 4 is a block diagram illustrating a processing unit of a CPU of the video management server. FIG. 5 is a diagram illustrating data stored in a video information DB. FIG. 6 is a diagram illustrating a detailed configuration of an image quality correction processing unit. FIG. 7 is a diagram illustrating correction auxiliary data. FIG. 8 is a diagram illustrating processing by the image quality correction processing unit when a registration request is received. FIG. 9 is a diagram illustrating processing by the image quality correction processing unit when a search request is received. FIG. 10 is a flowchart illustrating the processing procedure of an authentication unit. FIG. 11 is a diagram illustrating how the Euclidean distance in the feature amount space between a print-captured image and each marker image is derived as a similarity. FIG. 12 is a diagram illustrating a video playback display screen. FIG. 13 is a diagram illustrating a first notification screen. FIG. 14 is a flowchart illustrating the processing procedure of the video management server. FIG. 15 is a flowchart illustrating processing by the image quality correction processing unit of the second embodiment. FIG. 16 is a flowchart illustrating the processing procedure of the authentication unit of the second embodiment. FIG. 17 is a flowchart illustrating processing procedures of the image quality correction processing unit and the authentication unit of the second embodiment. FIG. 18 is a diagram illustrating processing by the RW control unit of the third embodiment. FIG. 10 is a diagram showing a process of assigning an identifier to a marker image. FIG. 11 is a diagram showing a third notification screen. FIG. 12 is a flowchart showing a processing procedure of a video management server of the third embodiment. FIG. 13 is a flowchart showing a processing procedure of a video management server of the third embodiment. FIG. 14 is a diagram showing a processing unit of the fourth embodiment. FIG. 15 is a diagram showing a method of registering a video and a marker image in the fifth embodiment. FIG. 16 is a diagram showing the processing of an image quality correction processing unit when a registration request is received in the fifth embodiment.

[0016] [First Embodiment] As an example, as shown in FIG. 1 , a user U owns a user terminal 10 and an instant camera 11. The user terminal 10 is a device having a camera function, an image playback / display function, an image editing function, an image transmission / reception function, and the like. The camera function has an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and obtains images of a subject (still images and videos 15) by focusing subject light captured through a lens on the imaging element. Specifically, the user terminal 10 is a digital camera, a smartphone, a tablet terminal, a notebook personal computer, or the like. The user U uses the camera function to take images and the image editing function to edit images to his or her liking.

[0017] The instant camera 11 is a device that has a camera function and a printer function. Unlike the user terminal 10, the camera function of the instant camera 11 does not have an image sensor, but rather obtains an image (still image) of a subject by focusing subject light captured through a lens onto the instant film 12. Alternatively, like the user terminal 10, the camera function of the instant camera 11 may have an image sensor and obtain an image of a subject by focusing subject light captured through a lens onto the image sensor. The printer function is a function that prints an image on the instant film 12. The instant film 12 may be either a silver halide type or a thermal type. The instant film 12 is an example of a "print medium" according to the technology of the present disclosure.

[0018] The user terminal 10 is connected to the video management server 14 via a network 13 so as to be able to communicate with each other. The network 13 is, for example, a WAN (Wide Area Network) such as the Internet or a public communication network. The user terminal 10 transmits (uploads) images to the video management server 14. The user terminal 10 also receives (downloads) images from the video management server 14.

[0019] Furthermore, the user terminal 10 is connected to the instant camera 11 via short-range wireless communication such as Bluetooth (registered trademark) so that they can communicate with each other. This allows the user to transfer images from the user terminal 10 to the instant camera 11 and print the transferred images on instant film 12 using the printer function of the instant camera 11.

[0020] The video management server 14 is, for example, a server computer, a workstation, or the like, and is an example of an “image processing device” according to the technology of the present disclosure. A plurality of user terminals 10 of a plurality of users U are connected to the video management server 14 via a network 13.

[0021] Here, a video distribution service using the user terminal 10 and instant camera 11 provided to the user U by the video management server 14 will be described with reference to the flowchart shown in Figure 2. First, the user U shoots a video 15 using the camera function of the user terminal 10 (step ST100). The video 15 is an example of "content" according to the technology of the present disclosure.

[0022] Next, the user U operates the user terminal 10 to register an arbitrary frame of the video 15 as a marker image 16 in the video management server 14 (step ST110). The video management server 14 associates the video 15 with the marker image 16 and registers the associated frame. The arbitrary frame is not particularly limited, but may be, for example, the first frame of the video 15. The marker image 16 serves as a so-called AR (Augmented Reality) marker.

[0023] After registering the marker image 16, the user U operates the user terminal 10 to transfer the marker image 16 from the user terminal 10 to the instant camera 11 (step ST120). Next, the user U operates the instant camera 11 to print the marker image 16 on the instant film 12 using the print function of the instant camera 11 (step ST130).

[0024] Thereafter, the user U uses the camera function of the user terminal 10 to photograph the instant film 12 on which the marker image 16 is printed, i.e., a print of the marker image 16 (step ST140). A photographed print image 17, which is an image obtained by photographing the print of the marker image 16, is transmitted from the user terminal 10 to the video management server 14. The photographed print image 17 is an example of a "photographed image" according to the technology of the present disclosure.

[0025] The video management server 14 searches for a marker image 16 corresponding to the print image 17 from the user terminal 10 (step ST150). The video 15 associated with the searched marker image 16 is then distributed from the video management server 14 to the user terminal 10 and played back and displayed on the user terminal 10 (step ST160). At this time, the print area 12A of the instant film 12 displayed on the user terminal 10 becomes the playback display area for the video 15, creating an effect as if the marker image 16 on the instant film 12 were a thumbnail of the video 15 (see FIG. 13). The video 15 is also tilted to match the angle at which the instant film 12 is displayed on the user terminal 10 (see FIG. 13). In this way, the video distribution service provided by the video management server 14 plays back and displays the video 15 associated with the marker image 16 when a print of the marker image 16 is captured. Note that reference numeral 12B in FIG. 1 indicates a frame area surrounding the print area 12A.

[0026] 3, the computers that make up the user terminal 10 and the video management server 14 basically have the same configuration, and include a storage 20, a memory 21, a CPU (Central Processing Unit) 22, a communication unit 23, a display 24, and an input device 25. These are interconnected via a bus line 26.

[0027] The storage 20 is a hard disk drive built into the computer that constitutes the user terminal 10 and the video management server 14, or connected via a cable or network. Alternatively, the storage 20 is a disk array consisting of multiple hard disk drives. The storage 20 stores control programs such as an operating system, various application programs (hereinafter abbreviated as APs (Application Programs)), and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0028] The memory 21 is a work memory for the CPU 22 to execute processing. The CPU 22 loads a program stored in the storage 20 into the memory 21 and executes processing in accordance with the program. In this way, the CPU 22 comprehensively controls each part of the computer. The CPU 22 is an example of a "processor" according to the technology of the present disclosure. The memory 21 may be built into the CPU 22.

[0029] The communication unit 23 is a network interface that controls the transmission of various information via the network 13, etc. The display 24 displays various screens. The various screens are equipped with operation functions using a GUI (Graphical User Interface). The computers that make up the user terminal 10 and the video management server 14 accept input of operation instructions from an input device 25 via the various screens. The input device 25 is a keyboard, mouse, touch panel, microphone for voice input, etc.

[0030] In the following explanation, the parts of the computer that make up the user terminal 10 (storage 20, CPU 22, display 24, and input device 25) are distinguished by adding the suffix "A" to their symbols, and the parts of the computer that make up the video management server 14 (storage 20 and CPU 22) are distinguished by adding the suffix "B" to their symbols.

[0031] As an example, as shown in Figure 4, a video AP 30 is stored in the storage 20A of the user terminal 10. The video AP 30 is installed in the user terminal 10 by the user U. The video AP 30 is an AP for enjoying the video distribution services shown in Figures 1 and 2 on the user terminal 10. When the video AP 30 is started, the CPU 22A of the user terminal 10 functions as a browser control unit 32 in cooperation with the memory 21 and the like. The browser control unit 32 controls the operation of a web browser dedicated to the video AP 30.

[0032] The browser control unit 32 receives screen data for various screens from the video management server 14. The browser control unit 32 reproduces various screens to be displayed on the web browser based on the screen data and displays them on the display 24A. The browser control unit 32 also accepts various operation instructions input by the user U from the input device 25A via the various screens. The browser control unit 32 transmits various requests in response to the operation instructions to the video management server 14. The browser control unit 32 also transmits a print request for the marker image 16 to the instant camera 11 in response to an instruction to print the marker image 16. The print request includes the marker image 16.

[0033] 5, an operating program 35 is stored in storage 20B of video management server 14. Operating program 35 is an AP that causes a computer constituting video management server 14 to function as an "image processing device" according to the technology of the present disclosure. In other words, operating program 35 is an example of an "operating program for an image processing device" according to the technology of the present disclosure.

[0034] The storage 20B also stores a video information database (hereinafter referred to as DB (Data Base)) 36, auxiliary correction data 37, a first setting range 38, etc. Although not shown in the figure, the storage 20B also stores, as account information of the user U, a user ID (Identification Data) for uniquely identifying the user U, a password set by the user U, and a terminal ID for uniquely identifying the user terminal 10.

[0035] When the operating program 35 is started, the CPU 22B of the video management server 14 cooperates with the memory 21 and the like to function as a request receiving unit 45, an image quality correction processing unit 46, a read / write (hereinafter abbreviated as RW (Read Write)) control unit 47, and a distribution control unit 48. The RW control unit 47 includes an authentication unit 50.

[0036] The request receiving unit 45 receives various requests from the user terminal 10. The request receiving unit 45 outputs various requests to the image quality correction processing unit 46 and / or the RW control unit 47 and the distribution control unit 48. The various requests include a registration request 75 (see FIG. 9) for registering the video 15 and the marker image 16 in the video information DB 36, and a search request 80 (see FIG. 10) for searching for the marker image 16 corresponding to the print-captured image 17. The various requests include the terminal ID of the user terminal 10 that sent the various requests.

[0037] Here, the marker image 16 is one frame of a video 15 captured using the camera function of the user terminal 10. In contrast, the print image 17 is an image captured from the instant film 12 on which the marker image 16 is printed. For this reason, there is a difference in image quality between the marker image 16 and the print image 17. This difference in image quality may hinder the search for the marker image 16 corresponding to the print image 17, resulting in erroneous authentication. If erroneous authentication occurs, problems may occur, such as the desired video 15 not being played back or a completely irrelevant video 15 different from the desired video 15 being played back.

[0038] Therefore, the image quality correction processing unit 46 performs image quality correction processing on the marker image 16 of the registration request 75 and the print image 17 of the search request 80 to reduce the difference in image quality between the marker image 16 and the print image 17. The image quality correction processing unit 46 outputs the marker image 16 and the print image 17 after the image quality correction processing to the RW control unit 47. Here, reducing the difference in image quality not only means eliminating the difference in image quality but also includes reducing the difference in image quality.

[0039] The RW control unit 47 controls the storage of various data in the storage 20 B and the reading of various data from the storage 20 B. In particular, the RW control unit 47 controls the registration of the video 15 and the marker image 16 in the video information DB 36 in response to a registration request 75, and the reading of the video 15 and the marker image 16 from the video information DB 36 in response to a search request 80.

[0040] The RW control unit 47 reads the correction auxiliary data 37 from the storage 20B and outputs the read correction auxiliary data 37 to the image quality correction processing unit 46. The RW control unit 47 also reads the first setting range 38 from the storage 20B and outputs the read first setting range 38 to the authentication unit 50.

[0041] In response to the search request 80, the authentication unit 50 searches for the marker image 16 corresponding to the photographed print image 17. The authentication unit 50 outputs the video 15 associated with the searched marker image 16 to the distribution control unit 48.

[0042] The distribution control unit 48 controls the distribution of various data including screen data to the user terminal 10. The distribution control unit 48 distributes the screen data to the user terminal 10 in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). Note that instead of XML, other data description languages ​​such as JSON (Javascript (registered trademark) Object Notation) may be used.

[0043] As an example, as shown in Fig. 6, the video information DB 36 has a storage area 55 for each user U, such as user U1 and user U2. A user ID is stored in the storage area 55. A pair of a video 15 and a marker image group 56 is associated and registered in the storage area 55. The marker image group 56 is a collection of multiple types of marker images 16 (see Fig. 9). Hereinafter, the pair of the video 15 and the marker image group 56 will be referred to as video information 57.

[0044] 7 , the image quality correction processing unit 46 includes a film type processing unit 60, a discoloration processing unit 61, and an ambient light removal unit 62. The film type processing unit 60 processes the marker image 16 according to the type of instant film 12. The discoloration processing unit 61 processes the marker image 16 according to discoloration caused by aging of the print of the marker image 16. The ambient light removal unit 62 removes the influence of ambient light when the print of the marker image 16 is photographed from the print image 17.

[0045] As an example, as shown in FIG. 8 , the correction auxiliary data 37 includes a three-dimensional lookup table for color film (hereinafter referred to as a 3D (Dimension)-LUT (Look Up Table)) 65 and a 3D-LUT for monochrome film 66. The 3D-LUT for color film 65 and the 3D-LUT for monochrome film 66 are used in the film type adaptation unit 60. The 3D-LUT for color film 65 is a 3D-LUT for reproducing the color of color instant film 12. The 3D-LUT for monochrome film 66 is a 3D-LUT for reproducing the color of monochrome instant film 12. The 3D-LUT converts the red pixel value, green pixel value, and blue pixel value of each pixel of the input image into the desired red pixel value, green pixel value, and blue pixel value, and outputs them.

[0046] The correction auxiliary data 37 also includes a 3D-LUT 67 for one-year fading correction, a 3D-LUT 68 for three-year fading correction, and a 3D-LUT 69 for five-year fading correction. The 3D-LUT 67 for one-year fading correction, the 3D-LUT 68 for three-year fading correction, and the 3D-LUT 69 for five-year fading correction are used in the fading processing unit 61. The 3D-LUT 67 for one-year fading correction, the 3D-LUT 68 for three-year fading correction, and the 3D-LUT 69 for five-year fading correction are 3D-LUTs for reproducing fading of a print of the marker image 16 over one year, three years, and five years. Note that, although one year, three years, and five years are given as examples here, the present invention is not limited to these. It may be every year for 1 to 10 years, or every 3 months, 6 months, 9 months, 1 year, 2 years, 4 years, 8 years, etc.

[0047] Furthermore, the correction auxiliary data 37 includes an ambient light removal filter 70. The ambient light removal filter 70 is used by the ambient light removal unit 62. As the name suggests, the ambient light removal filter 70 is a filter for removing the influence of ambient light when a print of the marker image 16 is photographed from the photographed print image 17. Multiple types of ambient light removal filters 70 are prepared according to the type of ambient light. The types of ambient light include sunny weather, cloudy weather, incandescent light, fluorescent light, flash, etc.

[0048] 9 , when a user U issues an instruction to register a video 15 and a marker image 16 via an input device 25A, the browser control unit 32 issues a registration request 75 to the video management server 14. The registration request 75 includes the user ID, the video 15, and the marker image 16.

[0049] The request receiving unit 45 receives the registration request 75. The request receiving unit 45 outputs the moving image 15 of the registration request 75 to the RW control unit 47 and the marker image 16 to the image quality correction processing unit 46.

[0050] The film type adaptation unit 60 of the image quality correction processing unit 46 applies a 3D-LUT 65 for color film to the marker image 16, converting the marker image 16 into a marker image 16A for color film. The marker image 16A for color film is a digitally reproduced image of the marker image 16 printed on color instant film 12. The film type adaptation unit 60 also applies a 3D-LUT 66 for monochrome film to the marker image 16, converting the marker image 16 into a marker image 16B for monochrome film. The marker image 16B is a digitally reproduced image of the marker image 16 printed on monochrome instant film 12. The film type adaptation unit 60 outputs the marker image 16A for color film to a fading processing unit 61.

[0051] The fading processing unit 61 applies a one-year fading correction 3D-LUT 67, a three-year fading correction 3D-LUT 68, and a five-year fading correction 3D-LUT 69 to the color film marker image 16A, respectively, and converts the color film marker image 16A into a one-year fading marker image 16C, a three-year fading marker image 16D, and a five-year fading marker image 16E. The one-year fading marker image 16C, the three-year fading marker image 16D, and the five-year fading marker image 16E are marker images 16 printed on color instant film 12, and are digital reproductions of the marker images 16 one year, three years, and five years after printing, respectively.

[0052] The image quality correction processing unit 46 outputs the original marker image 16, the color film corresponding marker image 16A, the monochrome film corresponding marker image 16B, the one-year fading corresponding marker image 16C, the three-year fading corresponding marker image 16D, and the five-year fading corresponding marker image 16E to the RW control unit 47 as a marker image group 56. The RW control unit 47 registers the pair of the moving image 15 from the request receiving unit 45 and the marker image group 56 from the image quality correction processing unit 46, i.e., moving image information 57, in a storage area 55 corresponding to the user ID of the registration request 75.

[0053] 10 , when a user U takes a picture of an instant film 12 using the camera function of the user terminal 10, the browser control unit 32 issues a search request 80 to the video management server 14. The search request 80 includes the user ID and the print image 17.

[0054] The request receiving unit 45 receives the search request 80. The request receiving unit 45 outputs the print photographed image 17 of the search request 80 to the image quality correction processing unit .

[0055] The ambient light removal unit 62 of the image quality correction processing unit 46 inputs the print image 17 to the ambient light removal filter 70. The ambient light removal filter 70 then outputs the ambient light-removed print image 17R. More specifically, the ambient light removal unit 62 extracts the print area 12A and the frame area 12B from the print image 17. The ambient light removal unit 62 identifies the type of ambient light from the pixel values ​​of the frame area 12B. The ambient light removal unit 62 then selects an ambient light removal filter 70 corresponding to the identified type and applies it to the print image 17. The ambient light removal filter 70 selectively performs ambient light removal processing on the print area 12A. The ambient light removal unit 62 outputs the ambient light-removed print image 17R to the authentication unit 50.

[0056] The authentication unit 50 performs authentication according to the processing procedure shown in the flowchart of FIG. 11 , for example. First, the authentication unit 50 reads all video information 57 registered in the storage area 55 corresponding to the user ID of the search request 80. Then, the authentication unit 50 derives features of the ambient light-removed print image 17R and each of the marker images 16 and 16A-16E in the marker image group 56 of all the read video information 57 (step ST3201). The features are, for example, a collection of representative pixel values ​​(average, mode, maximum, minimum, etc.), sharpness, blur, degree of defocus, and features obtained by inputting the marker images 16 and 16A-16E into a machine learning model such as an autoencoder. In other words, the features can be considered multidimensional feature vectors. Note that, in addition to the above, the features may also include a numerical value representing the type of main subject (person, animal, still life, landscape, etc.), the area occupied by the main subject, a numerical value corresponding to the position of the main subject, and the color of the main subject. Here, the trained model may be generated by supervised learning, such as the exemplary autoencoder, or by unsupervised learning. The trained model may output one or more features. The features output from the trained model are semantically incomprehensible to humans. However, any numerical value that is uniquely output when an image is input to the trained model can be used as a feature representing the image's characteristics. For the marker images 16 and 16A-16E, the feature values ​​may be derived in advance when the marker image group 56 is registered in the video information DB 36, and the derived feature values ​​may be associated with the marker image group 56 and stored in the video information DB 36.

[0057] The authentication unit 50 derives the similarity between the ambient light-removed print image 17R and each of the marker images 16 and 16A to 16E in a round-robin manner (step ST3202). More specifically, as shown in graph 85 of FIG. 12 as an example, the authentication unit 50 derives the Euclidean distance D in a feature vector space (hereinafter referred to as "feature space") 86 between the feature vector of the ambient light-removed print image 17R and the feature vector of each of the marker images 16 and 16A to 16E as the similarity. The Euclidean distance D can be used as an index representing the similarity between the ambient light-removed print image 17R and each of the marker images 16 and 16A to 16E. In other words, the closer the Euclidean distance D in the feature space 86 between the ambient light-removed print image 17R and each of the marker images 16 and 16A to 16E, the more similar the ambient light-removed print image 17R is to each of the marker images 16 and 16A to 16E. Conversely, the farther the Euclidean distance D in the feature space 86 between the ambient light-removed print image 17R and each of the marker images 16 and 16A to 16E, the less similar the ambient light-removed print image 17R is to each of the marker images 16 and 16A to 16E. Note that, for convenience of explanation, in FIG. 12 , the feature space 86 is represented two-dimensionally along the D1 axis and the D2 axis, but the actual dimension of the feature space 86 is several hundred to several thousand. Furthermore, the similarity may be derived as the Mahalanobis distance between the feature vector of the ambient light-removed print image 17R and the feature vector of each of the marker images 16 and 16A to 16E, instead of the Euclidean distance D. Alternatively, the similarity may be derived as the cosine similarity between the feature vector of the ambient light-removed print image 17R and the feature vector of each of the marker images 16 and 16A to 16E.

[0058] The authentication unit 50 compares the derived similarity with the first set range 38 (step ST3203). If there is a marker image 16 (hereinafter referred to as marker image 16X (see FIG. 10)) whose similarity to the ambient light-removed print image 17R falls within the first set range 38 (YES in step ST3203), the authentication unit 50 outputs the video 15 associated with the marker image 16X to the distribution control unit 48 (step ST3204). In this case, the distribution control unit 48 distributes screen data of the video playback display screen 90 (see FIG. 13) to the user terminal 10 that sent the search request 80. Note that reference numeral ST320A corresponds to step ST320A in FIG. 16.

[0059] 13, the browser control unit 32 receives screen data for a video playback display screen 90, recreates the video playback display screen 90 from the screen data, and displays it on the display 24A. On the video playback display screen 90, the print area 12A of the instant film 12 is treated as a playback display area, and the video 15 is played back and displayed.

[0060] On the other hand, if there is no marker image 16X whose similarity to the ambient light-removed print image 17R is within the first set range 38 (NO in step ST3203), the authentication unit 50 outputs to the distribution control unit 48 a message indicating that it was unable to find a corresponding video 15. In this case, the distribution control unit 48 distributes screen data of a first notification screen 95 (see FIG. 14 ) to the user terminal 10 that sent the search request 80.

[0061] 14, the browser control unit 32 receives screen data for a first notification screen 95, recreates the first notification screen 95 from the screen data, and displays it on the display 24A. The first notification screen 95 displays a message 96 indicating that the corresponding video 15 could not be found, and a confirmation button 97. When the confirmation button 97 is selected, the display of the first notification screen 95 is cleared.

[0062] If there are multiple marker images 16X within the first set range 38, the marker image 16X having the highest similarity to the ambient light removed print image 17R (the Euclidean distance D of the feature vector is closest) is used.

[0063] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 15 and 16 as an example. As shown in Figure 4, the CPU 22A of the user terminal 10 functions as a browser control unit 32 when the video AP 30 is started. As shown in Figure 5, the CPU 22B of the video management server 14 functions as a request receiving unit 45, an image quality correction processing unit 46, a RW control unit 47, and a distribution control unit 48 when the operating program 35 is started. The RW control unit 47 includes an authentication unit 50.

[0064] 1 and 2, a user U shoots a video 15 using the camera function of the user terminal 10. After shooting the video, the user U designates any one frame of the video 15 as a marker image 16. Then, as shown in FIG. 9, under the control of the browser control unit 32, a registration request 75 including the video 15 and the marker image 16 is sent to the video management server 14.

[0065] In the video management server 14, the registration request 75 is received by the request receiving unit 45 (YES in step ST200 of FIG. 15). The registration request 75 is output from the request receiving unit 45 to the image quality correction processing unit 46 and the like.

[0066] 9 , in the image quality correction processing unit 46, the film type processing unit 60 applies a color film 3D-LUT 65 and a monochrome film 3D-LUT 66 to the marker image 16. This converts the marker image 16 into a color film marker image 16A and a monochrome film marker image 16B. Furthermore, the fading processing unit 61 applies a one-year fading correction 3D-LUT 67, a three-year fading correction 3D-LUT 68, and a five-year fading correction 3D-LUT 69 to the color film marker image 16A. This converts the color film marker image 16A into a one-year fading marker image 16C, a three-year fading marker image 16D, and a five-year fading marker image 16E (step ST210). Each marker image 16 and a marker image group 56 consisting of marker images 16A to 16E are output from the image quality correction processing unit 46 to the RW control unit 47.

[0067] Under the control of the RW control unit 47, the video 15 and the marker image group 56, that is, the video information 57, are registered in the storage area 55 corresponding to the user ID in the registration request 75 (step ST220).

[0068] 1 and 2, a user U transfers a marker image 16 from a user terminal 10 to an instant camera 11. The marker image 16 is then printed on an instant film 12 using the print function of the instant camera 11.

[0069] The user U takes a photograph of a print of the marker image 16 using the camera function of the user terminal 10. A search request 80 including the thus obtained print image 17 is transmitted to the video management server 14 under the control of the browser control unit 32, as shown in FIG.

[0070] In the video management server 14, the search request 80 is received by the request receiving unit 45 (YES in step ST300 in FIG. 16). The search request 80 is output from the request receiving unit 45 to the image quality correction processing unit 46 and the like.

[0071] 10 , in the image quality correction processing unit 46, the print image 17 is input to the ambient light removal filter 70 in the ambient light removal unit 62. As a result, the ambient light removed print image 17R is output from the ambient light removal filter 70 (step ST310). The ambient light removed print image 17R is output from the image quality correction processing unit 46 to the authentication unit 50.

[0072] As shown in Figures 11 and 12, the authentication unit 50 searches for the marker image 16X corresponding to the ambient light-removed print image 17R (step ST320A). If the marker image 16X is found, the video 15 associated with the marker image 16X is output from the authentication unit 50 to the distribution control unit 48 (YES in step ST330). Then, under the control of the distribution control unit 48, screen data for the video playback display screen 90 shown in Figure 13 is distributed to the user terminal 10 that sent the search request 80 (step ST340). The video playback display screen 90 is displayed on the display 24A by the browser control unit 32. This allows the video 15 to be viewed by the user U.

[0073] On the other hand, if the marker image 16X is not found, the authentication unit 50 outputs to the distribution control unit 48 a message indicating that the corresponding video 15 could not be found (NO in step ST330). Then, under the control of the distribution control unit 48, screen data of the first notification screen 95 shown in FIG. 14 is distributed to the user terminal 10 that sent the search request 80. The first notification screen 95 is displayed on the display 24A by the browser control unit 32. This notifies the user U that the corresponding video 15 could not be found.

[0074] As described above, the request receiving unit 45 receives a search request 80 to acquire the print image 17 of the marker image 16, which is associated with the video 15 and triggers output of the video 15. The image quality correction processing unit 46 performs image quality correction processing on the marker image 16 and the print image 17 to reduce the difference in image quality between the marker image 16 and the print image 17. This reduces the risk of erroneous authentication caused by differences in image quality between the marker image 16 and the print image 17. This reduces the risk of problems such as the desired video 15 not being played or a completely irrelevant video 15 being played and displayed.

[0075] 9 and 10, the authentication unit 50 searches for the marker image 16X that has been subjected to the image quality correction process and that corresponds to the print image 17 that has been subjected to the image quality correction process. This can fully demonstrate the effect of reducing the risk of erroneous authentication caused by the difference in image quality between the marker image 16 and the print image 17.

[0076] Furthermore, the authentication unit 50 outputs the video 15 associated with the searched marker image 16X to the distribution control unit 48. This makes it possible to fully exert the effect of reducing the risk of erroneous authentication occurring due to a difference in image quality between the marker image 16 and the print-captured image 17.

[0077] 9, the film type adaptation unit 60 of the image quality correction processing unit 46 performs image quality correction processing by applying processing according to the type of instant film 12 used to print the marker image 16. This reduces the risk of erroneous authentication occurring due to differences in the type of instant film 12.

[0078] 9, the fading processing unit 61 of the image quality correction processing unit 46 performs image quality correction processing to apply processing in response to fading of the print of the marker image 16 due to aging. This reduces the risk of erroneous authentication caused by fading of the print of the marker image 16 due to aging.

[0079] 10, the ambient light removal unit 62 of the image quality correction processing unit 46 performs image quality correction processing to remove the influence of ambient light when a print of the marker image 16 is photographed. This reduces the risk of erroneous authentication caused by ambient light.

[0080] 6, the RW control unit 47 associates the moving image 15 and the marker image 16 and registers them in the moving image information DB 36. Therefore, the moving image 15 associated with the searched marker image 16X can be immediately output.

[0081] Instant film 12 is popular among young people because, compared to regular photographic film, which takes time to develop, it can be developed on the spot, allowing for easy sharing with friends, and it can create a retro atmosphere that is difficult to achieve with digital images. For this reason, instant film 12 is ideal for the video distribution service of the present disclosure, which boasts an appealing presentation. On the other hand, instant film 12 has poor color reproducibility and inferior image quality compared to photographic film. Therefore, using instant film 12 as the printing medium for printing marker image 16 can enhance the advantage of reducing the risk of erroneous authentication due to differences in image quality between marker image 16 and printed captured image 17.

[0082] 17 , an image quality correction processing unit 46 of the second embodiment includes a difference extraction unit 100 and a noise removal / interpolation unit 101 in addition to the processing units 60 to 62 of the first embodiment. The difference extraction unit 100 and the noise removal / interpolation unit 101 operate when there is no marker image 16X whose similarity to the ambient light-removed print image 17R falls within the first set range 38, but there is a marker image 16Y whose similarity falls within a second set range 105 (see FIG. 18 ) that is looser than the first set range 38. If the first set range 38 is, for example, a similarity less than 10, the second set range 105 is, for example, a similarity greater than or equal to 10 and less than 50.

[0083] A case where marker image 16X is absent but marker image 16Y is present may occur when a stamp object 102, such as a star mark added by user U when taking a print of marker image 16, is captured in ambient light-removed print image 17R, as shown in the figure. Stamp object 102 is an example of “noise” according to the technology of the present disclosure. In addition to the illustrated stamp object 102, noise may also include character objects such as animated characters, as well as traces of various effect processing, such as a light leak effect that reproduces the light leak that occurs on actual instant film. Other examples of noise include stickers attached by user U to the print of marker image 16, illustrations and / or messages actually handwritten by user U using an oil-based pen or the like on the print of marker image 16, and illustrations and / or messages electronically drawn by user U on the print of marker image 16 using a drawing function provided by video AP 30. Further noise may include scratches and / or dust on the instant film 12, the user's hand holding the instant film 12, and / or other instant film 12 or other obstructions.

[0084] The difference extraction unit 100 calculates the difference between the marker image 16Y and the print area 12A of the ambient light-removed print image 17R, and extracts a difference image 103. If the stamp object 102, which is noise, is removed from the ambient light-removed print image 17R, and the marker image 16Y is the image searched for as the marker image 16X, only the stamp object 102, which is noise, will remain in the difference image 103, as shown in the illustrated example. The difference extraction unit 100 outputs the difference image 103 to the noise removal / interpolation unit 101.

[0085] In addition to the difference image 103, the noise removal / interpolation unit 101 receives the ambient light-removed print image 17R and a noise removal / interpolation model 104. The noise removal / interpolation model 104 is stored in the storage 20B. The noise removal / interpolation model 104 is a machine learning model constructed using a convolutional neural network or the like. The noise removal / interpolation model 104 has a function of removing noise represented by the difference image 103 from the ambient light-removed print image 17R and interpolating the removed noise with the pixel values ​​of surrounding pixels. The noise removal / interpolation unit 101 inputs the ambient light-removed print image 17R and the difference image 103 to the noise removal / interpolation model 104, and causes the noise removal / interpolation model 104 to output a noise-removed / interpolated print image 17DN (see FIG. 18 ). The noise removal / interpolation unit 101 outputs the noise-removed / interpolated print image 17DN to the authentication unit 50.

[0086] 18, a marker image 16Y and a noise-removed and interpolated print image 17DN are input to the authentication unit 50. In addition to the first set range 38, a second set range 105 is also input to the authentication unit 50. The second set range 105 is stored in the storage 20B.

[0087] As an example, the authentication unit 50 performs authentication according to the processing procedure shown in step ST320B of the flowcharts in Figures 19 and 20. The processing of step ST320B is performed instead of the processing of step ST320A in the first embodiment. Note that the same step numbers are assigned to the same processing procedures as in the first embodiment.

[0088] First, as in the first embodiment, the authentication unit 50 reads all moving image information 57 stored in the storage area 55 corresponding to the user ID of the search request 80, and derives features of the ambient light-removed print image 17R and each of the marker images 16 and 16A to 16E in the marker image group 56 of the read moving image information 57 (step ST3201). Next, the authentication unit 50 derives the similarity between the ambient light-removed print image 17R and each of the marker images 16 and 16A to 16E (step ST3202).

[0089] The authentication unit 50 compares the derived similarity with the first set range 38 (step ST3203). If there is a marker image 16X whose similarity with the ambient light-removed printed photographed image 17R is within the first set range 38 (YES in step ST3203), the authentication unit 50 outputs the video 15 associated with the marker image 16X to the distribution control unit 48 (step ST3204).

[0090] On the other hand, if there is no marker image 16X whose similarity to the ambient light-removed print image 17R is within the first set range 38 (NO in step ST3203), the authentication unit 50 compares the derived similarity with the second set range 105 (step ST3205).If there is a marker image 16Y whose similarity to the ambient light-removed print image 17R is within the second set range 105 (YES in step ST3205), the difference extraction unit 100 extracts a difference image 103 between the marker image 16Y and the ambient light-removed print image 17R (step ST3206), as shown in FIG. 17, and outputs the difference image 103 to the noise removal / interpolation unit 101. The noise removal / interpolation unit 101 removes noise from the ambient light removed print image 17R using the noise removal / interpolation model 104 and performs interpolation to convert the ambient light removed print image 17R into a noise removed / interpolated print image 17DN (step ST3207). The noise removal / interpolation unit 101 outputs the noise removed / interpolated print image 17DN to the authentication unit 50.

[0091] The authentication unit 50 derives the similarity between the noise-removed and interpolated photographed print image 17DN and the marker image 16Y (step ST3208). The authentication unit 50 compares the derived similarity with the first set range 38 (step ST3209). If the similarity between the noise-removed and interpolated photographed print image 17DN and the marker image 16Y is within the first set range 38 (YES in step ST3209), the authentication unit 50 outputs the video 15 associated with the marker image 16Y to the distribution control unit 48 (step ST3204).

[0092] On the other hand, if there is no marker image 16Y whose similarity with the ambient light-removed printed image 17R is within the second setting range 105 (NO in step ST3205), and if the similarity between the noise-removed and interpolated printed image 17DN and the marker image 16Y is not within the first setting range 38 (NO in step ST3209), the authentication unit 50 outputs to the distribution control unit 48 a message that it was unable to find a corresponding video 15.

[0093] As described above, in the second embodiment, the difference extraction unit 100 and the noise removal / interpolation unit 101 of the image quality correction processing unit 46 perform processing to remove and interpolate noise that is present in a print of the marker image 16, such as the exemplified stamp object 102, and that interferes with searching for the marker image 16 that corresponds to the captured print image 17. This reduces the risk of erroneous authentication caused by noise.

[0094] The marker image 16X and the ambient light-removed print image 17R may be subjected to the difference extraction process and noise removal / interpolation process described above. In this case, if the similarity between the marker image 16X and the ambient light-removed print image 17R becomes lower than before the noise removal / interpolation process, the marker image 16X is adopted. However, if the similarity becomes higher, the marker image 16X is not adopted.

[0095] 21 , a third setting range 110 is input to the RW control unit 47 of the third embodiment. The third setting range 110 is stored in the storage 20B. The third setting range 110 is an example of a “setting range” according to the technology of the present disclosure.

[0096] When the RW control unit 47 receives a registration request 75 from the request receiving unit 45, the RW control unit 47 derives a similarity between the marker image 16 to be registered included in the registration request 75 and all marker images 16 already registered in the storage area 55 of the video information DB 36 corresponding to the user ID of the registration request 75 through a round-robin search. The RW control unit 47 compares the derived similarity with a third set range 110. If there is a registered marker image 16 whose similarity falls within the third set range 110, the RW control unit 47 controls the distribution control unit 48 to distribute screen data of a second notification screen 115 (see FIG. 22 ) to the user terminal 10 that transmitted the registration request 75.

[0097] 22 as an example, the browser control unit 32 receives screen data for a second notification screen 115, reproduces the second notification screen 115 from the screen data, and displays it on the display 24A. The second notification screen 115 displays a message 116 informing the user that a marker image 16 similar to the marker image 16 to be registered has already been registered, and asking the user U whether or not to register the marker image 16 to be registered with an identifier 120 (see FIG. 23). The second notification screen 115 also displays a Yes button 117 and a No button 118.

[0098] When the user U selects the Yes button 117, the browser control unit 32 issues an identifier addition request to the video management server 14. The request receiving unit 45 receives the identifier addition request and outputs it to the RW control unit 47. As shown in FIG. 23 as an example, the RW control unit 47 assigns an identifier 120, such as a QR (Quick Response) Code (registered trademark), to each of the marker images 16 and 16A to 16E constituting the marker image group 56. The marker image group 56 to which the identifier 120 has been assigned is then associated with the video 15 and registered in the video information DB 36. When the identifier 120 has been assigned in this manner, the identifier 120 is also printed on the instant film 12 together with the marker image 16. Note that FIG. 23 illustrates an example in which the identifier 120 is assigned to the marker image 16.

[0099] On the other hand, if the No button 118 is selected by the user U, the RW control unit 47 uses the distribution control unit 48 to distribute the screen data of the third notification screen 125 (see Figure 24) to the user terminal 10 that sent the registration request 75.

[0100] 24 as an example, the browser control unit 32 receives screen data for a third notification screen 125, recreates the third notification screen 125 from the screen data, and displays it on the display 24A. The third notification screen 125 displays a message 126 informing the user U that the marker image 16 to be registered cannot be registered and urging the user U to try registering another marker image 16. The third notification screen 125 also displays a confirmation button 127. When the confirmation button 127 is selected, the display of the third notification screen 125 is cleared.

[0101] 25 and 26 show an example of a processing procedure of the video management server 14 of the third embodiment. Note that the same processing procedures as those of the first embodiment are assigned the same step numbers.

[0102] After the registration request 75 is received by the request receiving unit 45 (YES in step ST200), the RW control unit 47 derives the similarity between the marker image 16 to be registered included in the registration request 75 and all marker images 16 already registered in the memory area 55 of the video information DB 36 corresponding to the user ID of the registration request 75 (step ST230).

[0103] The RW control unit 47 compares the derived similarity with the third set range 110 (step ST240). If there is no registered marker image 16 whose similarity falls within the third set range 110 (NO in step ST240), the film type processing unit 60 and the fading processing unit 61 of the image quality correction processing unit 46 process the marker image 16 in the registration request 75 according to the type of instant film 12 and the fading, respectively, as in the first embodiment (step ST210). Then, under the control of the RW control unit 47, the moving image 15 and the marker image group 56, i.e., the moving image information 57, are registered in the storage area 55 corresponding to the user ID in the registration request 75 (step ST220).

[0104] On the other hand, if there is a registered marker image 16 whose similarity is within the third setting range 110 (YES in step ST240), under the control of the distribution control unit 48, the screen data of the second notification screen 115 shown in Figure 22 is distributed to the user terminal 10 that sent the registration request 75, thereby displaying the second notification screen 115 on the display 24A of the user terminal 10.

[0105] When the Yes button 117 on the second notification screen 115 is selected and the request for adding an identifier from the browser control unit 32 is received by the request receiving unit 45 (YES in step ST250), the image quality correction processing unit 46 performs image quality correction processing (step ST210), and then the RW control unit 47 assigns the identifier 120 to each of the marker images 16 and 16A to 16E constituting the marker image group 56 (step ST260), as shown in Fig. 23. Thereafter, under the control of the RW control unit 47, the video 15 and the marker image group 56, i.e., the video information 57, are registered in the storage area 55 corresponding to the user ID in the registration request 75 (step ST220).

[0106] If the No button 118 on the second notification screen 115 is selected (NO in step ST250), under the control of the distribution control unit 48, the screen data of the third notification screen 125 shown in Figure 24 is distributed to the user terminal 10 that sent the registration request 75, and the third notification screen 125 is thereby displayed on the display 24A of the user terminal 10.

[0107] As described above, in the third embodiment, the RW control unit 47 derives the similarity between the marker image 16 to be registered and the already registered marker image 16. If there is a registered marker image 16 whose similarity with the marker image 16 to be registered is within the third setting range 110, the RW control unit 47 does not register the marker image 16 to be registered. Alternatively, the RW control unit 47 registers the marker image 16 to be registered with an identifier 120 to distinguish it from the already registered marker image 16. This reduces the risk of erroneous authentication caused by a similar marker image 16.

[0108] The request receiving unit 45 receives requests such as an identifier addition request in response to an instruction from the user U as to whether or not to add the identifier 120 to the marker image 16 to be registered. Therefore, the intention of the user U can be reflected in the registration of the marker image 16.

[0109] If there is a registered marker image 16 whose similarity to the marker image 16 to be registered is within the third set range 110, the marker image 16 to be registered may not be registered without waiting for selection by the user U. Similarly, if there is a registered marker image 16 whose similarity to the marker image 16 to be registered is within the third set range 110, the marker image 16 to be registered may be assigned an identifier 120 and registered without waiting for selection by the user U.

[0110] [Fourth Embodiment] As shown in FIG. 27 as an example, the CPU 22B of the video management server 14 of the fourth embodiment functions as a film type identification unit 130 in addition to the processing units 45 to 48 and 50 (only the authentication unit 50 is shown in FIG. 27 ) of the first embodiment. A print image 17 is input to the film type identification unit 130. The film type identification unit 130 extracts a print area 12A and a frame area 12B from the print image 17. The film type identification unit 130 identifies the type of instant film 12 from which the print image 17 was taken based on pixel values ​​in the frame area 12B. The film type identification unit 130 then outputs an identification result 131 including the identified type to the authentication unit 50. The authentication unit 50 searches for a marker image 16X by limiting the search to the type corresponding to the identification result 131. For example, if the identification result 131 is a monochrome instant film 12, the authentication unit 50 searches for the marker image 16X by limiting the search to the monochrome film-compatible marker image 16B. This reduces the time required to search for the marker image 16X.

[0111] Alternatively, the photographing date and time printed on the print photographed image 17 may be character-recognized, and the marker image 16X may be searched for within a limited period of time corresponding to the character-recognized photographing date and time. For example, if the character-recognized photographing date and time is three years ago, the search for the marker image 16X may be limited to the three-year fading compatible marker image 16D. This also reduces the time required to search for the marker image 16X.

[0112] [Fifth Embodiment] In each of the above embodiments, an example has been given in which a single user U enjoys a video distribution service, but this is not limiting. As an example, as shown in FIG. 28 , a plurality of users U may enjoy a video distribution service. Here, two users UA and UB are exemplified as the plurality of users U. Users UA and UB are acquaintances such as relatives, friends, and coworkers. User UA transfers a video 15 that he or she shot to user UB. User UA also transfers to user UB an instant film 12 on which he or she printed a marker image 16 using an instant camera 11 (a print of the marker image 16).

[0113] User UB imports the video 15 transferred from user UA into his own user terminal 10B. User UB also uses the camera function of his own user terminal 10B to photograph the print of marker image 16 transferred from user UA in order to register the marker image 16. Hereinafter, the image photographed of the print of marker image 16 in order to register the marker image 16 will be referred to as marker image 16P. User UB uses the browser control unit 32 to issue a registration request 75 for the imported video 15 and the photographed marker image 16P to the video management server 14.

[0114] As shown in FIG. 29 as an example, in the fifth embodiment, when the request receiving unit 45 receives a registration request 75, the ambient light removal unit 62 operates in addition to the film type processing unit 60 and the discoloration processing unit 61. The ambient light removal unit 62 inputs the marker image 16P in the registration request 75 to the ambient light removal filter 70. The ambient light removal filter 70 then outputs an ambient light-removed marker image 16PR. More specifically, the ambient light removal unit 62 extracts the print area 12A and the frame area 12B from the marker image 16P. The ambient light removal unit 62 identifies the type of ambient light from the pixel values ​​of the frame area 12B. The ambient light removal unit 62 then selects an ambient light removal filter 70 corresponding to the identified type and applies it to the marker image 16P. The ambient light removal filter 70 selectively performs ambient light removal processing on the print area 12A. The ambient light removal unit 62 outputs the ambient light removed marker image 16PR to the film type adaptation unit 60. The subsequent processing is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0115] The reference symbols 16PRA and 16PRB in the marker image group 56 indicate color film-compatible marker images and monochrome film-compatible marker images output by applying the color film 3D-LUT 65 and the monochrome film 3D-LUT 66 to the ambient light-removed marker image 16PR in the film type compatibility processing unit 60. The reference symbols 16PRC, 16PRD, and 16PRE indicate one-year fading-compatible marker images, three-year fading-compatible marker images, and five-year fading-compatible marker images output by applying the one-year fading correction 3D-LUT 67, the three-year fading correction 3D-LUT 68, and the five-year fading correction 3D-LUT 69 to the color film-compatible marker image 16PRA in the fading processing unit 61.

[0116] In this way, in the fifth embodiment, the marker images 16 printed on the instant film 12 exchanged between users U are registered. This allows a plurality of users U to enjoy the video distribution service.

[0117] Furthermore, the ambient light removal unit 62 performs processing to remove the influence of ambient light from the marker image 16P, which is an image obtained by photographing a print of the marker image 16. This reduces the risk of erroneous authentication caused by ambient light.

[0118] The image quality correction processing unit 46 may include at least one of the exemplified film type adaptation processing unit 60, discoloration processing unit 61, and ambient light removal unit 62. For example, the ambient light removal unit 62 may be omitted, and the image quality correction processing unit 46 may be configured by the film type adaptation processing unit 60 and the discoloration processing unit 61. In this case, the image quality correction processing is performed only on the marker image 16, and the image quality correction processing is not performed on the print captured image 17.

[0119] In addition, if the video 15 is stored in the storage 20A of the user terminal 10, instead of outputting the video 15 associated with the searched marker image 16X, a playback display instruction signal for the video 15 associated with the searched marker image 16X may be output.

[0120] Although two examples of processing according to the type of instant film 12 have been given, namely, processing corresponding to color instant film 12 and processing corresponding to monochrome instant film 12, this is not limiting. Only one of processing corresponding to color instant film 12 and processing corresponding to monochrome instant film 12 may be used.

[0121] The print medium is not limited to the instant film 12 shown in the example. It may be L-size or other photo paper. The captured image is not limited to the print image 17 captured on the instant film 12 shown in the example. It may be a captured image of the marker image 16 displayed on a display device such as the display 24A of the user terminal 10.

[0122] Although the video 15 is given as an example of the content, the content is not limited to this. It may be 3D computer graphics of products such as automobiles, furniture, and clothing. It may also be 3D computer graphics for tourist guides created by local governments.

[0123] The hardware configuration of the computer that constitutes the video management server 14 can be modified in various ways. For example, the video management server 14 can be configured with multiple computers separated as hardware in order to improve processing power and reliability. For example, the functions of the request reception unit 45 and the image quality correction processing unit 46 and the functions of the RW control unit 47 and the distribution control unit 48 can be distributed and performed by two computers. In this case, the video management server 14 is configured with two computers. In addition, all or part of the functions of the video management server 14 may be performed by the user terminal 10.

[0124] In this way, the hardware configuration of the computer of the user terminal 10 and the video management server 14 can be changed as appropriate according to the required performance such as processing power, safety, reliability, etc. Furthermore, it is not limited to hardware, but APs such as the video AP 30 and the operating program 35 can also be duplicated or stored in multiple storages in order to ensure safety and reliability.

[0125] In each of the above embodiments, the hardware structure of the processing units that perform various processes, such as the browser control unit 32, request reception unit 45, image quality correction processing unit 46, RW control unit 47, distribution control unit 48, authentication unit 50, film type adaptation processing unit 60, fading processing unit 61, ambient light removal unit 62, difference extraction unit 100, noise removal / interpolation unit 101, and film type identification unit 130, can be any of the various processors listed below. The various processors include CPUs 22A and 22B, which are general-purpose processors that execute software (video AP 30 and operating program 35) and function as various processing units, as well as programmable logic devices (PLDs), which are processors whose circuit configuration can be changed after manufacture, such as FPGAs (Field Programmable Gate Arrays), and / or dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors having a circuit configuration designed specifically for executing specific processing.

[0126] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0127] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0128] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0129] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0130] [Supplementary Item 1] An image processing device including a processor, which acquires a captured image of a marker image associated with content and which triggers output of the content, and performs image quality correction processing on at least one of the marker image or the captured image to reduce a difference in image quality between the marker image and the captured image. [Supplementary Item 2] The image processing device according to Supplementary Item 1, wherein the processor searches for the marker image corresponding to the captured image on which the image quality correction processing has been performed, the marker image corresponding to the captured image on which the image quality correction processing has been performed, or the marker image corresponding to the captured image on which the image quality correction processing has been performed. [Supplementary Item 3] The image processing device according to Supplementary Item 2, wherein the processor outputs the content associated with the searched marker image. [Supplementary Item 4] The image processing device according to any one of Supplementary Items 1 to 3, wherein the captured image is a captured print image of a print of the marker image, and the processor performs, as the image quality correction process, at least one of: processing according to the type of print medium used to print the marker image, processing according to fading of the print of the marker image over time, removing the influence of ambient light when the print of the marker image was photographed, and removing and interpolating noise present in the print of the marker image that interferes with searching for the marker image corresponding to the captured image. [Supplementary Item 5] The image processing device according to any one of Supplementary Items 1 to 4, wherein the processor registers the content and the marker image in association with each other. [Supplementary Item 6] The image processing device according to Supplementary Item 5, wherein the processor derives a similarity between a marker image to be registered and a registered marker image, and if there is a registered marker image whose similarity with the marker image to be registered is within a set range, does not register the marker image to be registered, or registers the marker image to be registered with an identifier attached thereto to distinguish it from the registered marker image.[Supplementary Item 7] The image processing device according to Supplementary Item 6, wherein the processor receives a user instruction as to whether or not to register the marker image to be registered with the identifier. [Supplementary Item 8] The image processing device according to any one of Supplementary Items 1 to 7, wherein the captured image is a print image obtained by photographing a print of the marker image, and a print medium used to print the marker image is instant film.

[0131] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs, but also to storage media that non-temporarily store programs, and computer program products that include programs.

[0132] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0133] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0134] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising a processor, which acquires a captured image of a marker image associated with content that triggers output of the content, and performs image quality correction processing on at least one of the marker image or the captured image to reduce the difference in image quality between the marker image and the captured image.

2. The image processing device described in claim 1, wherein the processor searches for the marker image corresponding to the captured image on which the image quality correction processing has been performed, the marker image corresponding to the captured image on which the image quality correction processing has been performed, or the marker image corresponding to the captured image on which the image quality correction processing has been performed and on which the image quality correction processing has been performed.

3. The image processing device according to claim 2, wherein the processor outputs the content associated with the searched marker image.

4. The image processing device according to claim 1, wherein the captured image is a print image of a print of the marker image, and the processor performs, as the image quality correction process, at least one of the following processes: a process of applying processing appropriate to the type of print medium used to print the marker image; a process of applying processing appropriate to fading of the print of the marker image over time; a process of removing the effects of ambient light when the print of the marker image was photographed; and a process of removing and interpolating noise present in the print of the marker image that interferes with searching for the marker image corresponding to the captured image.

5. The image processing device according to claim 1, wherein the processor registers the content and the marker image in association with each other.

6. The image processing device according to claim 5, wherein the processor derives a similarity between a marker image to be registered and a registered marker image, and if there is a registered marker image whose similarity with the marker image to be registered is within a set range, does not register the marker image to be registered, or registers the marker image to be registered with an identifier to distinguish it from the registered marker image.

7. The image processing device according to claim 6, wherein the processor receives an instruction from a user as to whether or not to register the marker image to be registered with the identifier.

8. The image processing device according to claim 1, wherein the captured image is a print image of a print of the marker image, and the print medium used to print the marker image is instant film.

9. A method for operating an image processing device, comprising: acquiring a captured image of a marker image associated with content, the marker image being a trigger for outputting the content; and performing image quality correction processing on at least one of the marker image or the captured image to reduce the difference in image quality between the marker image and the captured image.

10. An operating program for an image processing device that causes a computer to execute a process including: acquiring a captured image of a marker image associated with content that triggers the output of said content; and performing image quality correction processing on at least one of said marker image or said captured image to reduce the difference in image quality between said marker image and said captured image.

Images