Image processing apparatus, method of operating an image processing apparatus, and program for operating an image processing apparatus

Abstract

An image processing apparatus includes a processor configured to acquire a captured image obtained by capturing a marker image associated with content and serving as an output trigger of the content, and perform a quality correction process on at least one of the marker image or the captured image to reduce a difference in quality between the marker image and the captured image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing apparatus operating method, and an image processing apparatus operating procedure. Background Technology

[0002] The content management system described in Japanese Patent Application Publication No. 2016-058024 is a content management system for managing content composed of managed images and motion image data associated with the managed images. It includes a still image data extraction unit, a person of interest detection unit, a motion trajectory detection unit, a motion analysis unit, a still image data output unit, a management tag storage unit, and a management image generation unit. The still image data extraction unit extracts still image data from multiple frames of motion image data. The person of interest detection unit detects the person of interest, i.e., the object of processing, from each of the multiple still images corresponding to the multiple frames of still image data. The motion trajectory detection unit detects the motion trajectory of the person of interest by tracking the movement of the person of interest in the motion image corresponding to the motion image data, based on the detection results of the person of interest in the multiple still images. The motion analysis unit analyzes the actions of the person of interest in the motion image based on the motion trajectory of the person of interest, and calculates an evaluation value for the actions of the person of interest for each of the multiple still images based on the analyzed actions of the person of interest. The still image data output unit outputs one still image from multiple frames of still image data, whose evaluation value for the action of the person of interest is above a threshold, as output still image data. The management tag storage unit associates and stores the image feature values ​​of the output still image data or the corresponding output still image data with the motion image data of the motion image corresponding to the output still image data as management tags. The management image generation unit generates management image data including the output still image data.

[0003] The content reproduction system described in Japanese Patent Application Publication No. 2016-001821 includes a server and a mobile terminal connected to the server via a network. The server includes a storage unit that, for each of one or more moving image data sets, associates selected image data (which is selected from one or more still image data sets extracted from the moving image data) or specific information of the selected image data as a management tag corresponding to the selected image data with the moving image data and stores it. The mobile terminal includes an image capturing unit that captures and outputs an image of the selected image to obtain captured image data. The server further includes: an image analysis unit that performs image analysis on the captured image data to obtain a management tag for the captured image corresponding to the captured image data; a management tag determination unit that determines a management tag for the selected image corresponding to the management tag of the captured image from the management tags of the selected image stored in the storage unit as a specific management tag; and a summary moving image generation unit that extracts a portion of the moving image data associated with the specific management tag, i.e., related moving image data, to generate summary moving image data. The mobile terminal also includes: a display unit that reproduces and displays a summary dynamic image corresponding to the summary dynamic image data; and a control unit that, when an output image is captured by an image capturing unit, controls the display unit to reproduce and display a summary dynamic image corresponding to the summary dynamic image data generated based on the captured image data. Summary of the Invention

[0004] One embodiment of the present invention provides an image processing apparatus, an image processing apparatus operation method, and an image processing apparatus operation procedure that can reduce the possibility of false authentication caused by the difference in image quality between a marker image that reproduces content starting from the capture and a captured image that captured the marker image.

[0005] means for solving technical problems

[0006] The image processing apparatus of the present invention includes a processor that performs the following processing: acquiring a captured image, which is obtained by capturing a marker image that is associated with content and becomes a marker image for outputting content; and performing image quality correction processing on at least one of the marker image or the captured image to reduce the image quality difference between the marker image and the captured image.

[0007] The preferred processor performs the following processing: retrieves a marker image corresponding to the captured image that has undergone image quality correction processing, a marker image corresponding to the captured image that has undergone image quality correction processing, or a marker image corresponding to the captured image that has undergone image quality correction processing that has undergone image quality correction processing.

[0008] The preferred processor output is content that is associated with the retrieved tagged image.

[0009] The preferred image to be captured is a printout of a printed document containing the marking image. The processor performs at least one of the following processing as image quality correction processing: processing corresponding to the type of printing medium used for the printout of the marking image, processing corresponding to the fading of the printout of the marking image over the years, processing to remove the influence of ambient light when the printout of the marking image was captured, and processing to remove noise present in the printout of the marking image that obstructs the retrieval of the marking image corresponding to the captured image and to perform interpolation.

[0010] The preferred processor performs the following processing: exporting the similarity between the tag image to be registered and the registered tag images; and if there is a registered tag image whose similarity to the tag image to be registered is within a set range, either not registering the tag image to be registered, or attaching an identifier to the tag image to be registered and registering it, the identifier being used to distinguish it from the registered tag images.

[0011] The preferred processor receives instructions from the user regarding whether to attach an identifier to the tagged image to be registered and then register it.

[0012] The preferred processor performs the following processing: establishing an association between the content and the tagged image and registering them.

[0013] Preferably, the image captured is an image of a printout of a printout containing the marked image, and the printing medium for the printout containing the marked image is instant imaging film.

[0014] The image processing apparatus of the present invention includes the following steps: acquiring a captured image, which is obtained by capturing a marker image that is associated with content and becomes the output opportunity of content; and performing image quality correction processing on at least one of the marker image or the captured image to reduce the image quality difference between the marker image and the captured image.

[0015] The operating procedure of the image processing apparatus of the present invention causes a computer to perform a process including the following steps: acquiring a captured image, which is obtained by capturing a marker image that is associated with content and becomes the output opportunity of the content; and performing image quality correction processing on at least one of the marker image or the captured image to reduce the image quality difference between the marker image and the captured image. Attached Figure Description

[0016] Figure 1 This diagram represents the user terminal, the instant camera, and the dynamic image management server.

[0017] Figure 2 It is a flowchart representing the process from capturing a moving image on the user terminal to reproducing and displaying the moving image on the user terminal.

[0018] Figure 3 This is a block diagram representing the computer that constitutes the user terminal and the dynamic image management server.

[0019] Figure 4 This is a block diagram representing the CPU processing unit of a user terminal.

[0020] Figure 5 This is a block diagram representing the CPU processing unit of the dynamic image management server.

[0021] Figure 6 It is a graph representing the data stored in the dynamic image information DB.

[0022] Figure 7 This is a diagram showing the detailed structure of the image quality correction processing unit.

[0023] Figure 8 It is a graph representing the correction auxiliary data.

[0024] Figure 9 This diagram illustrates the processing by the image quality correction department when a registration request is received.

[0025] Figure 10 This diagram illustrates the processing by the image quality correction unit when a retrieval request is received.

[0026] Figure 11 This is a flowchart illustrating the processing steps of the certification department.

[0027] Figure 12 It is a graph representing the state derived from the Euclidean distance in the feature space between the printed image and each marked image as the similarity.

[0028] Figure 13 It is a diagram that represents the display of a dynamic image.

[0029] Figure 14 This is an image representing the first notification screen.

[0030] Figure 15 This is a flowchart illustrating the processing steps of the dynamic image management server.

[0031] Figure 16 This is a flowchart illustrating the processing steps of the dynamic image management server.

[0032] Figure 17 This diagram illustrates the processing of the image quality correction processing unit in the second embodiment.

[0033] Figure 18 This diagram illustrates the processing of the authentication unit in the second embodiment.

[0034] Figure 19This is a flowchart illustrating the processing steps of the authentication unit in the second embodiment.

[0035] Figure 20 This is a flowchart illustrating the processing steps of the image quality correction processing unit and the authentication unit in the second embodiment.

[0036] Figure 21 This is a diagram showing the processing of the RW control unit in the third embodiment.

[0037] Figure 22 This is an image representing the second notification screen.

[0038] Figure 23 This is a diagram illustrating the process of attaching identifiers to labeled images.

[0039] Figure 24 This is an image representing the third notification screen.

[0040] Figure 25 This is a flowchart illustrating the processing steps of the dynamic image management server in the third embodiment.

[0041] Figure 26 This is a flowchart illustrating the processing steps of the dynamic image management server in the third embodiment.

[0042] Figure 27 This is a diagram showing the processing unit of the fourth embodiment.

[0043] Figure 28 This is a diagram illustrating the registration method for dynamic images and labeled images in the fifth embodiment.

[0044] Figure 29 This diagram illustrates the processing of the image quality correction processing unit when a registration request is received in the fifth embodiment. Detailed Implementation

[0045] [First Implementation]

[0046] As an example, such as Figure 1 As shown, user U has user terminal 10 and instant camera 11. User terminal 10 is a device with camera function, image reproduction and display function, image editing function, and image transmission and reception function. The camera function is a function that uses an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor to obtain an image of the subject (still image and moving image 15) by imaging the light of the subject taken in through a lens onto the imaging element. Specifically, user terminal 10 is a digital camera, smartphone, tablet terminal, or laptop computer, etc. User U uses the camera function to take pictures or uses the image editing function to edit the images into their preferred style.

[0047] The instant camera 11 is a device with both camera and printer functions. Unlike the user terminal 10, the camera function of the instant camera 11 is to obtain an image (still image) of the subject by imaging the light of the subject captured through a lens onto the instant imaging film 12, without having an imaging element. Alternatively, the camera function of the instant camera 11 can also be the same as that of the user terminal 10, having an imaging element to obtain an image of the subject by imaging the light of the subject captured through a lens onto the imaging element. The printer function is to print the image onto the instant imaging film 12. The instant imaging film 12 can be either silver halide or thermal. The instant imaging film 12 is an example of a "printing medium" involved in the technology of this invention.

[0048] User terminal 10 is communicatively connected to moving image management server 14 via network 13. Network 13 is, for example, the Internet or a public communication network (WAN). User terminal 10 sends (uploads) images to moving image management server 14. Furthermore, user terminal 10 receives (downloads) images from moving image management server 14.

[0049] Furthermore, the user terminal 10 can communicate with the instant camera 11 via short-range wireless communication such as Bluetooth (registered trademark). Therefore, it is possible to transmit images from the user terminal 10 to the instant camera 11 and use the printer function of the instant camera 11 to print the transmitted images onto the instant imaging film 12.

[0050] The dynamic image management server 14 is, for example, a server computer, a workstation, etc., and is an example of the "image processing apparatus" involved in the technology of this invention. Multiple user terminals 10 of multiple users U are connected to the dynamic image management server 14 via a network 13.

[0051] Here, as an example, for reference Figure 2 The flowchart shown illustrates the video transmission service provided by the video management server 14 to user U, which utilizes user terminal 10 and instant camera 11. First, user U uses the camera function of user terminal 10 to capture video 15 (step ST100). Video 15 is an example of the "content" involved in the technology of this invention.

[0052] Next, user U operates user terminal 10 to register any frame of motion image 15 as marker image 16 to motion image management server 14 (step ST110). In motion image management server 14, motion image 15 and marker image 16 are associated and registered. Furthermore, the "any frame" is not particularly limited; for example, it can be the first frame of motion image 15. Marker image 16 functions as a so-called AR (Augmented Reality) marker.

[0053] After registering the marker image 16, user U operates user terminal 10 to transfer the marker image 16 from user terminal 10 to instant camera 11 (step ST120). Then, user U operates instant camera 11 to print the marker image 16 on instant imaging film 12 using the printing function of instant camera 11 (step ST130).

[0054] Then, user U uses the camera function of user terminal 10 to photograph the instant imaging film 12 with the marked image 16 printed on it, i.e., the printout of the marked image 16 (step ST140). The printout photographed image 17, which is the image obtained by photographing the printout of the marked image 16, is sent from user terminal 10 to dynamic image management server 14. The printout photographed image 17 is an example of the "photographed image" involved in the technology of this invention.

[0055] In the motion image management server 14, a marker image 16 corresponding to the printed image 17 from the user terminal 10 is retrieved (step ST150). Then, the motion image 15 associated with the retrieved marker image 16 is distributed from the motion image management server 14 to the user terminal 10 and reproduced on the user terminal 10 (step ST160). At this time, the printed area 12A of the instant imaging film 12 displayed on the user terminal 10 becomes the reproduction display area of ​​the motion image 15, and a demonstration is performed as if the marker image 16 of the instant imaging film 12 is a thumbnail of the motion image 15 (see reference). Figure 13 Furthermore, a demonstration is given of displaying a dynamic image 15 based on the shooting angle of the real-time imaging film 12 displayed on the user terminal 10 (see reference). Figure 13 Thus, the dynamic image transmission service provided by the dynamic image management server 14, taking the opportunity of capturing a printout of the marked image 16, reproduces and displays the dynamic image 15 associated with the marked image 16. Furthermore, Figure 1 The symbol 12B represents the border area surrounding the printed area 12A.

[0056] As an example, such as Figure 3As shown, the computers constituting the user terminal 10 and the dynamic image management server 14 have essentially the same structure, including a storage device 20, a memory 21, a CPU (Central Processing Unit) 22, a communication unit 23, a display 24, and an input device 25. They are connected to each other via a bus 26.

[0057] Storage device 20 is a hard disk drive built into the computer constituting user terminal 10 and dynamic image management server 14 or connected via cable or network. Alternatively, storage device 20 is a hard disk array consisting of multiple hard disk drives connected together. Storage device 20 stores control programs such as operating systems, various application programs (hereinafter referred to as APs), and various data associated with these programs. Alternatively, solid-state drives can be used instead of hard disk drives.

[0058] Memory 21 is a working memory used by CPU 22 for performing processing. CPU 22 loads the program stored in storage device 20 into memory 21 and executes the processing according to the program. Thus, CPU 22 centrally controls the various parts of the computer. CPU 22 is an example of a "processor" according to the technology of this invention. In addition, memory 21 may be built into CPU 22.

[0059] The communication unit 23 is a network interface for controlling the transmission of various information via network 13, etc. The display 24 shows various screens. These screens have GUI (Graphical User Interface) based operation functions. The computer constituting the user terminal 10 and the dynamic image management server 14 receives operation instructions from the input device 25 through these screens. The input device 25 includes a keyboard, mouse, touch panel, and microphone for voice input, etc.

[0060] In addition, in the following description, the parts of the computer constituting the user terminal 10 (storage device 20, CPU 22, display 24 and input device 25) will be distinguished by the suffix "A" in the reference numerals, and the parts of the computer constituting the dynamic image management server 14 (storage device 20 and CPU 22) will be distinguished by the suffix "B" in the reference numerals.

[0061] As an example, such as Figure 4 As shown, a moving image AP30 is stored in the storage device 20A of the user terminal 10. The moving image AP30 is installed on the user terminal 10 by the user U. The moving image AP30 is for enjoying on the user terminal 10. Figure 1 and Figure 2The illustrated AP provides a dynamic image transmission service. When the dynamic image AP 30 is activated, the CPU 22A of the user terminal 10, in cooperation with the memory 21, functions as a browser control unit 32. The browser control unit 32 controls the operation of the dedicated web browser of the dynamic image AP 30.

[0062] The browser control unit 32 receives screen data from various screens in the dynamic image management server 14. Based on the screen data, the browser control unit 32 reproduces the various screens displayed on the web browser and displays them on the monitor 24A. Furthermore, the browser control unit 32 receives various operation instructions input by the user U from the input device 25A through various screens. The browser control unit 32 sends various requests corresponding to the operation instructions to the dynamic image management server 14. Additionally, based on the printing instruction for the marked image 16, the browser control unit 32 sends a printing request for the marked image 16 to the instant camera 11. The printing request includes the marked image 16.

[0063] As an example, such as Figure 5 As shown, a working program 35 is stored in the storage device 20B of the dynamic image management server 14. The working program 35 is an application program (AP) for enabling the computer constituting the dynamic image management server 14 to function as an "image processing apparatus" according to the technology of this invention. That is, the working program 35 is an example of a "working program of an image processing apparatus" according to the technology of this invention.

[0064] The storage device 20B also stores a dynamic image information database (hereinafter referred to as DB) 36, correction auxiliary data 37, and a first setting range 38, etc. Furthermore, although the illustration is omitted, the storage device 20B stores a user ID (Identification Data) used to uniquely identify the user U, a password set by the user U, and a terminal ID used to uniquely identify the user terminal 10, serving as the user U's account information.

[0065] If the working program 35 is started, the CPU 22B of the dynamic image management server 14, together with the memory 21, etc., functions as a request receiving unit 45, an image quality correction processing unit 46, a read / write (hereinafter referred to as RW) control unit 47, and a distribution control unit 48. The RW control unit 47 includes an authentication unit 50.

[0066] The request receiving unit 45 receives various requests from the user terminal 10. The request receiving unit 45 outputs various requests to the image quality correction processing unit 46 and / or the RW control unit 47 and the distribution control unit 48. Among the various requests is a registration request 75 for registering the moving image 15 and the marker image 16 in the moving image information DB36 (see reference). Figure 9 ), and a retrieval request 80 for retrieving the marker image 16 corresponding to the printed image 17 (see reference). Figure 10 ), etc. The various requests include the terminal ID of the user terminal 10 that sent the various requests.

[0067] Here, the marker image 16 is a frame of the moving image 15 captured using the camera function of the user terminal 10. In contrast, the printout image 17 is an image obtained by capturing an instant imaging film 12 with the marker image 16 printed on it. Therefore, there is a difference in image quality between the marker image 16 and the printout image 17. This difference in image quality may become an obstacle when retrieving the marker image 16 corresponding to the printout image 17, resulting in false authentication. In the event of false authentication, there is a possibility of undesirable situations such as the expected moving image 15 not being reproduced or a moving image 15 that is completely unrelated to the expected moving image 15 being reproduced.

[0068] Therefore, the image quality correction processing unit 46 performs image quality correction processing on the marker image 16 of the registration request 75 and the printed image 17 of the retrieval request 80 to reduce the image quality difference between the marker image 16 and the printed image 17. The image quality correction processing unit 46 outputs the image quality correction processed marker image 16 and printed image 17 to the RW control unit 47. Here, reducing the image quality difference means not only eliminating the image quality difference, but also reducing the image quality difference.

[0069] RW control unit 47 controls the storage of various data to storage device 20B and the reading of various data from storage device 20B. In particular, RW control unit 47 controls the registration of dynamic image 15 and marker image 16 to dynamic image information DB36 corresponding to registration request 75, and the reading of dynamic image 15 and marker image 16 from dynamic image information DB36 corresponding to retrieval request 80.

[0070] The RW control unit 47 reads the correction auxiliary data 37 from the storage device 20B and outputs the read correction auxiliary data 37 to the image quality correction processing unit 46. Furthermore, the RW control unit 47 reads the first setting range 38 from the storage device 20B and outputs the read first setting range 38 to the authentication unit 50.

[0071] The authentication unit 50 retrieves the marker image 16 corresponding to the printed image 17 based on the retrieval request 80. The authentication unit 50 outputs the dynamic image 15 associated with the retrieved marker image 16 to the distribution control unit 48.

[0072] The distribution control unit 48 controls the distribution of various data containing screen data to the user terminal 10. The distribution control unit 48 distributes the screen data to the user terminal 10, for example, in the form of web pages created using markup languages ​​such as XML (Extensible Markup Language). Alternatively, other data description languages ​​such as JSON (Javascript Object Notation) can be used instead of XML.

[0073] As an example, such as Figure 6 As shown, in the dynamic image information DB36, a storage area 55 is set up for each user U, such as user U1 and user U2. The user ID is stored in the storage area 55. The dynamic image 15 and the labeled image group 56 are registered in an associated manner in the storage area 55. The labeled image group 56 is a collection of various labeled images 16 (see reference). Figure 9 Hereinafter, the group of moving image 15 and the group of labeled image 56 will be labeled as moving image information 57.

[0074] As an example, such as Figure 7 As shown, the image quality correction processing unit 46 includes a film type matching processing unit 60, a fading processing unit 61, and an ambient light removal unit 62. The film type matching processing unit 60 performs processing on the marked image 16 to correspond to the type of the instant imaging film 12. The fading processing unit 61 performs processing on the marked image 16 to correspond to the fading of the printed copy of the marked image 16 over the years. The ambient light removal unit 62 removes the influence of ambient light from the printed copy image 17 to remove the printed copy image of the marked image 16.

[0075] As an example, such as Figure 8 As shown, the correction auxiliary data 37 includes a 3D-LUT 65 for color film and a 3D-LUT 66 for monochrome film. Both the 3D-LUT 65 for color film and the 3D-LUT 66 for monochrome film are used in the film type corresponding processing unit 60. The 3D-LUT 65 for color film is a 3D-LUT used to reproduce the hue of the color instant imaging film 12. The 3D-LUT 66 for monochrome film is a 3D-LUT used to reproduce the hue of the monochrome instant imaging film 12. Furthermore, the 3D-LUT is a 3D-LUT that converts the red, green, and blue pixel values ​​of each pixel in the input image into the desired red, green, and blue pixel values ​​and outputs them.

[0076] Furthermore, the correction auxiliary data 37 includes a 1-year fading correction 3D-LUT 67, a 3-year fading correction 3D-LUT 68, and a 5-year fading correction 3D-LUT 69. These 3D-LUTs are used in the fading processing section 61. The 1-year fading correction 3D-LUT 67, 3-year fading correction 3D-LUT 68, and 5-year fading correction 3D-LUT 69 are 3D-LUTs used to reproduce the fading of the printed copy of the marked image 16 after 1 year, 3 years, and 5 years, respectively. While 1 year, 3 years, and 5 years are exemplified here, the data is not limited to these. It can be annually from 1 to 10 years, or monthly, semi-annually, 9 months, 1 year, 2 years, 4 years, and 8 years, etc.

[0077] Furthermore, the correction auxiliary data 37 includes an ambient light removal filter 70. The ambient light removal filter 70 is used in the ambient light removal unit 62. As its name suggests, the ambient light removal filter 70 is a filter used to remove the influence of ambient light when capturing the printed mark image 16 from the printed image 17. Various ambient light removal filters 70 are prepared depending on the type of ambient light. Types of ambient light include sunny days, cloudy days, incandescent lamps, fluorescent lamps, and flash lamps, etc.

[0078] As an example, such as Figure 9 As shown, when user U issues a registration instruction for animated image 15 and marker image 16 via input device 25A, browser control unit 32 sends registration request 75 to animated image management server 14. Registration request 75 includes user ID, animated image 15, and marker image 16.

[0079] The request receiving unit 45 receives the registration request 75. The request receiving unit 45 outputs the dynamic image 15 of the registration request 75 to the RW control unit 47 and outputs the marker image 16 to the image quality correction processing unit 46.

[0080] The film type matching processing unit 60 of the image quality correction processing unit 46 applies a 3D-LUT 65 to the marking image 16 using color film, converting the marking image 16 into a color film-matching marking image 16A. The color film-matching marking image 16A is a digitally reproduced image of the marking image 16 printed on color instant imaging film 12. Furthermore, the film type matching processing unit 60 applies a 3D-LUT 66 to the marking image 16 using monochrome film, converting the marking image 16 into a monochrome film-matching marking image 16B. The monochrome film-matching marking image 16B is a digitally reproduced image of the marking image 16 printed on monochrome instant imaging film 12. The film type matching processing unit 60 outputs the color film-matching marking image 16A to the fading processing unit 61.

[0081] The fading processing unit 61 applies 3D-LUT 67 (1-year fading correction), 3D-LUT 68 (3-year fading correction), and 3D-LUT 69 (5-year fading correction) to the corresponding marking image 16A on the color film, converting the color film corresponding marking image 16A into a 1-year fading corresponding marking image 16C, a 3-year fading corresponding marking image 16D, and a 5-year fading corresponding marking image 16E. The 1-year fading corresponding marking image 16C, the 3-year fading corresponding marking image 16D, and the 5-year fading corresponding marking image 16E are marking images 16 printed on the color instant imaging film 12, and are digital reproduction images of the marking images 16 that have been printed for 1 year, 3 years, and 5 years, respectively.

[0082] The image quality correction processing unit 46 outputs the original marker image 16, the marker image 16A corresponding to color film, the marker image 16B corresponding to monochrome film, the marker image 16C corresponding to 1 year fading, the marker image 16D corresponding to 3 years fading, and the marker image 16E corresponding to 5 years fading as a marker image group 56 to the RW control unit 47. The RW control unit 47 registers the group of moving image 15 from the request receiving unit 45 and the marker image group 56 from the image quality correction processing unit 46, i.e., moving image information 57, in the storage area 55 corresponding to the user ID of the registration request 75.

[0083] As an example, such as Figure 10 As shown, when user U takes a live image 12 using the camera function of user terminal 10, browser control unit 32 sends a retrieval request 80 to dynamic image management server 14. Retrieval request 80 includes user ID and the printed image 17.

[0084] The request receiving unit 45 receives the retrieval request 80. The request receiving unit 45 then outputs the image 17 of the printout of the retrieval request 80 to the image quality correction processing unit 46.

[0085] The ambient light removal unit 62 of the image quality correction processing unit 46 inputs the printed image 17 to the ambient light removal filter 70. Then, it outputs the ambient light removed printed image 17R from the ambient light removal filter 70. More specifically, the ambient light removal unit 62 extracts the printing area 12A and the border area 12B from the printed image 17. The ambient light removal unit 62 determines the type of ambient light based on the pixel values ​​of the border area 12B. Then, it selects the ambient light removal filter 70 corresponding to the determined type and applies it to the printed image 17. The ambient light removal filter 70 selectively performs ambient light removal processing on the printing area 12A. The ambient light removal unit 62 outputs the ambient light removed printed image 17R to the authentication unit 50.

[0086] As an example, the certification department passed 50. Figure 11The authentication process is performed according to the flowchart shown. First, the authentication unit 50 reads all motion image information 57 registered in the storage area 55 corresponding to the user ID of the retrieval request 80. Then, it exports the feature quantities of each labeled image 16 and 16A to 16E of the labeled image group 56 of the ambient light removed printout image 17R and all the read motion image information 57 (step ST3201). The feature quantities are, for example, representative values ​​of pixel values ​​(mean, mode, maximum, minimum, etc.), sharpness, jitter, defocus, and a set of feature quantities obtained by inputting the labeled images 16 and 16A to 16E into a machine learning model such as an autoencoder. That is, the feature quantities can be said to be multi-dimensional feature vectors. In addition to the above, as feature quantities, they can also be values ​​representing the type of the main subject (people, animals, still life, landscape, etc.), the area occupied by the main subject, the values ​​corresponding to the position of the main subject, and the hue of the main subject. Here, the trained model can be a model generated by supervised learning such as the illustrated autoencoder, or a model generated by unsupervised learning. The feature quantity output from the trained model can be one or more. The feature quantity output from the trained model is one that cannot be semantically interpreted by humans. However, if the output value is a unique value when an image is input into the trained model, then a feature quantity representing the image's characteristics can be used. Regarding the labeled images 16 and 16A–16E, the feature quantity can be derived when registering the labeled image group 56 to the dynamic image information DB36, and the derived feature quantity can be associated with the labeled image group 56 and stored in the dynamic image information DB36.

[0087] The certification department 50 uses exhaustive comparison to derive the similarity between the ambient light removed printed image 17R and the marked images 16 and 16A-16E (step ST3202). More specifically, as an example, such as... Figure 12As shown in Figure 85, the authentication unit 50 derives the Euclidean distance D in the feature vector space (hereinafter referred to as the feature space) 86 between the feature vector of the ambient light removed printed image 17R and the feature vectors of each marked image 16 and 16A to 16E, as a similarity. The Euclidean distance D can be used as an indicator to represent the similarity between the ambient light removed printed image 17R and each marked image 16 and 16A to 16E. That is, it can be said that the smaller the Euclidean distance D in the feature space 86 between the ambient light removed printed image 17R and each marked image 16 and 16A to 16E, the more similar the ambient light removed printed image 17R is to each marked image 16 and 16A to 16E. Conversely, it can be said that the greater the Euclidean distance D in the feature space 86 between the ambient light removed printed image 17R and the marked images 16 and 16A-16E, the less similar the ambient light removed printed image 17R is to the marked images 16 and 16A-16E. Furthermore, in Figure 12 For ease of explanation, the feature space 86 of the D1 and D2 axes is represented in two dimensions, but the actual feature space 86 has dimensions of hundreds to thousands. Furthermore, the Mahalanobis distance between the feature vector of the ambient light removed printed image 17R and the feature vectors of the marked images 16 and 16A-16E can be derived as the similarity, instead of the Euclidean distance D. Alternatively, the cosine similarity between the feature vector of the ambient light removed printed image 17R and the feature vectors of the marked images 16 and 16A-16E can be derived as the similarity.

[0088] The authentication department 50 compares the exported similarity with the first set range 38 (step ST3203). The marked image 16 (hereinafter referred to as marked image 16X) whose similarity to the ambient light removed printed image 17R is within the first set range 38 is selected. Figure 10 In the case of (Yes in step ST3203), the authentication unit 50 outputs the motion image 15 associated with the marker image 16X to the distribution control unit 48 (step ST3204). In this case, the distribution control unit 48 displays the motion image on the display screen 90 (see reference). Figure 13 The image data is distributed to the user terminal 10 of the source that sent the retrieval request 80. Additionally, the symbol ST320A and... Figure 16 The steps correspond to ST320A.

[0089] As an example, such as Figure 13As shown, the browser control unit 32 receives the image data of the motion image reproduction display screen 90, reproduces the motion image reproduction display screen 90 based on the image data, and displays it on the monitor 24A. In the motion image reproduction display screen 90, the printing area 12A of the instant imaging film 12 is regarded as the reproduction display area, and the motion image 15 is reproduced and displayed.

[0090] On the other hand, if a marker image 16X with a similarity to the ambient light removed printed image 17R within the first set range 38 does not exist ("No" in step ST3203), the authentication unit 50 outputs the content of the failed-to-find moving image 15 to the distribution control unit 48. In this case, the distribution control unit 48 displays the first notification screen 95 (see reference). Figure 14 The image data is distributed to the user terminal 10 of the source of the retrieval request 80.

[0091] As an example, such as Figure 14 As shown, the browser control unit 32 receives the screen data from the first notification screen 95, reproduces the first notification screen 95 based on the screen data, and displays it on the monitor 24A. The first notification screen 95 displays a message 96 indicating that the corresponding animated image 15 could not be found, and an OK button 97. When the OK button 97 is selected, the first notification screen 95 is closed.

[0092] In addition, when there are multiple labeled images 16X within the first set range 38, the labeled image 16X with the highest similarity (closest to the Euclidean distance D of the feature vector) to the printed image 17R after ambient light removal is selected.

[0093] Next, as an example, refer to Figure 15 and Figure 16 The flowchart shown illustrates the function of the above structure. For example... Figure 4 As shown, the CPU 22A of the user terminal 10 functions as the browser control unit 32 upon activation of the dynamic image AP 30. Furthermore, as... Figure 5 As shown, the CPU 22B of the dynamic image management server 14 functions as a request receiving unit 45, an image quality correction processing unit 46, an RW control unit 47, and a distribution control unit 48 upon the activation of the working program 35. The RW control unit 47 includes an authentication unit 50.

[0094] like Figure 1 and Figure 2 As shown, user U uses the camera function of user terminal 10 to capture moving images 15. After capturing the moving images, user U designates any one frame of moving images 15 as a marker image 16. And, as... Figure 9As shown, under the control of the browser control unit 32, a registration request 75 including the animated image 15 and the marker image 16 is sent to the animated image management server 14.

[0095] In the dynamic image management server 14, the request receiving unit 45 receives the registration request 75. Figure 15 (Step ST200 is marked "Yes"). Registration request 75 is output from request receiving unit 45 to image quality correction processing unit 46, etc.

[0096] like Figure 9 As shown, in the image quality correction processing unit 46, in the film type corresponding processing unit 60, a 3D-LUT 65 for color film and a 3D-LUT 66 for monochrome film are applied to the marked image 16. Thus, the marked image 16 is converted into a marked image 16A for color film and a marked image 16B for monochrome film. Furthermore, in the fading processing unit 61, a 3D-LUT 67 for 1-year fading correction, a 3D-LUT 68 for 3-year fading correction, and a 3D-LUT 69 for 5-year fading correction are applied to the marked image 16A for color film. Thus, the marked image 16A for color film is converted into a marked image 16C for 1-year fading, a marked image 16D for 3-year fading, and a marked image 16E for 5-year fading (step ST210). The marked image group 56, composed of each marked image 16 and 16A to 16E, is output from the image quality correction processing unit 46 to the RW control unit 47.

[0097] Under the control of the RW control unit 47, the dynamic image 15 and the marked image group 56, i.e., dynamic image information 57, are registered in the storage area 55 corresponding to the user ID of the registration request 75 (step ST220).

[0098] like Figure 1 and Figure 2 As shown, user U transmits the marked image 16 from user terminal 10 to instant camera 11. Then, the marked image 16 is printed on instant imaging film 12 using the printing function of instant camera 11.

[0099] The user U uses the camera function of user terminal 10 to take a printout of the marked image 16. For example... Figure 10 As shown, a retrieval request 80 containing the printed image 17 thus obtained is sent to the dynamic image management server 14 under the control of the browser control unit 32.

[0100] In the dynamic image management server 14, the request receiving unit 45 receives the retrieval request 80 ( Figure 16 (In step ST300, it is "Yes"). The retrieval request 80 is output from the request receiving unit 45 to the image quality correction processing unit 46, etc.

[0101] like Figure 10 As shown, in the image quality correction processing unit 46, the ambient light removal unit 62 inputs the printed image 17 to the ambient light removal filter 70. As a result, the ambient light removed printed image 17R is output from the ambient light removal filter 70 (step ST310). The ambient light removed printed image 17R is then output from the image quality correction processing unit 46 to the authentication unit 50.

[0102] In the certification department 50, such as Figure 11 and Figure 12 As shown, a marker image 16X corresponding to the ambient light removed printed image 17R is retrieved (step ST320A). If the marker image 16X is retrieved, a dynamic image 15 associated with the marker image 16X is output from the authentication unit 50 to the distribution control unit 48 ("Yes" in step ST330). Then, under the control of the distribution control unit 48, Figure 13 The image data of the animated image display screen 90 is distributed to the user terminal 10, the source of the retrieval request 80 (step ST340). The animated image display screen 90 is displayed on the monitor 24A by the browser control unit 32. Thus, the animated image 15 is made available for user U to view.

[0103] On the other hand, if the marker image 16X is not found, the content of the corresponding animated image 15 is not found and is output from the authentication unit 50 to the distribution control unit 48 ("No" in step ST330). Then, under the control of the distribution control unit 48, Figure 14 The screen data of the first notification screen 95 shown is distributed to the user terminal 10, the source of the retrieval request 80. The first notification screen 95 is displayed on the display 24A by the browser control unit 32. Thus, the user U is notified that the content of the corresponding moving image 15 could not be retrieved.

[0104] As explained above, the request receiving unit 45 receives the retrieval request 80 to obtain the printed image 17 of the marker image 16, which is associated with the moving image 15 and serves as the output trigger for the moving image 15. The image quality correction processing unit 46 performs image quality correction processing on the marker image 16 and the printed image 17 to reduce the image quality difference between the marker image 16 and the printed image 17. Therefore, the possibility of false authentication due to the image quality difference between the marker image 16 and the printed image 17 can be reduced. The possibility of adverse situations such as failure to reproduce the desired moving image 15 or reproduction of a moving image 15 that is completely unrelated to the desired moving image 15 can be reduced.

[0105] like Figure 9 and Figure 10As shown, the authentication unit 50 retrieves a marker image 16X that has undergone image quality correction processing, corresponding to the printout image 17 that has undergone image quality correction processing. Therefore, it is possible to fully reduce the possibility of false authentication due to the difference in image quality between the marker image 16X and the printout image 17.

[0106] Furthermore, the authentication unit 50 outputs a dynamic image 15 associated with the retrieved marker image 16X to the distribution control unit 48. Therefore, it is possible to fully reduce the possibility of false authentication due to differences in image quality between the marker image 16 and the printed image 17.

[0107] like Figure 9 As shown, the image quality correction processing unit 46 performs the following processing on the film type corresponding to the processing unit 60: as image quality correction processing, processing is performed corresponding to the type of instant imaging film 12 used for marking the printout of image 16. Therefore, the possibility of misidentification due to different types of instant imaging film 12 can be reduced.

[0108] like Figure 9 As shown, the fading processing unit 61 of the image quality correction processing unit 46 performs the following processing: as image quality correction processing, it performs processing corresponding to the fading of the printed copy of the marking image 16 over the years. Therefore, it is possible to reduce the possibility of false authentication due to the fading of the printed copy of the marking image 16 over the years.

[0109] like Figure 10 As shown, the ambient light removal unit 62 of the image quality correction processing unit 46 performs the following processing: as an image quality correction process, it removes the influence of ambient light when the printed copy of the marking image 16 was captured. Therefore, it is possible to reduce the possibility of false authentication due to ambient light.

[0110] like Figure 6 As shown, the RW control unit 47 associates the motion image 15 and the marker image 16 and registers them in the motion image information DB36. Therefore, the motion image 15 associated with the retrieved marker image 16X can be output immediately.

[0111] Compared to conventional film, which requires time to develop, instant imaging film 12 offers the ease of immediate development and enjoyment with companions, or the nostalgic atmosphere difficult to experience in digital images, thus gaining popularity primarily among young people. Therefore, incorporating instant imaging film 12 into the dynamic image delivery service of this invention, which is marketed as an engaging presentation, is preferable. On the other hand, instant imaging film 12 has poor color reproduction and image quality compared to film. Therefore, when instant imaging film 12 is used as the printing medium for the printed document used to mark the image 16, its superiority in reducing the possibility of misidentification due to image quality differences between the marked image 16 and the printed document image 17 can be enhanced.

[0112] [Second Implementation]

[0113] As an example, such as Figure 17 As shown, the image quality correction processing unit 46 of the second embodiment, in addition to the processing units 60 to 62 of the first embodiment described above, also includes a difference extraction unit 100 and a noise removal / interpolation unit 101. The difference extraction unit 100 and the noise removal / interpolation unit 101 operate within a second setting range 105 (see reference) where the similarity to the image 17R of the printed document after ambient light removal is within a wider range than the first setting range 38. Figure 18 Processing is performed if the marked image 16Y is within the first set range 38 but the marked image 16X is not within the first set range 38. If the first set range 38 is, for example, less than 10 in similarity, the second set range 105 is, for example, more than 10 in similarity and less than 50.

[0114] As in the case where there is no marker image 16X but there is a marker image 16Y, as shown in the figure, it is possible that a stamp object 102, such as a star-shaped mark, attached by the user U when photographing the printout of the marker image 16 is reflected in the printout image 17R after ambient light removal. The stamp object 102 is an example of "noise" involved in the technology of this invention. In addition, as noise, besides the illustrated stamp object 102, there are traces of various effect processing, such as character objects such as animated characters and light leakage effect processing that reproduces light leakage generated in actual real-time imaging film. Furthermore, as noise, there are stickers pasted by the user U on the printout of the marker image 16, illustrations and / or messages actually handwritten by the user U on the printout of the marker image 16 with an oil-based pen, and illustrations and / or messages electronically drawn by the user U on the printout of the marker image 16 using the drawing function provided in the motion picture AP30. In addition, as noise, there are scratches and / or dust on the instant imaging film 12, the user's hand holding the instant imaging film 12 and / or other obstructions such as the instant imaging film 12.

[0115] The differential extraction unit 100 extracts the difference between the marked image 16Y and the printed area 12A in the ambient light removed print image 17R, thereby extracting these differential images 103. If the stamp object 102, which is noise, is removed from the ambient light removed print image 17R, and if the marked image 16Y is an image that can be retrieved as the marked image 16X, then, as shown in the example, only the stamp object 102, which is noise, remains in the differential image 103. The differential extraction unit 100 outputs the differential image 103 to the noise removal / interpolation unit 101.

[0116] In addition to the difference image 103, the ambient light removed printout image 17R and the noise removal / interpolation model 104 are also input to the noise removal / interpolation unit 101. The noise removal / interpolation model 104 is stored in the storage device 20B. The noise removal / interpolation model 104 is a machine learning model built based on convolutional neural networks, etc. The noise removal / interpolation model 104 has the following functions: removing noise represented by the difference image 103 from the ambient light removed printout image 17R, and interpolating the removed noise using the pixel values ​​of the surrounding pixels. The noise removal / interpolation unit 101 inputs the ambient light removed printout image 17R and the difference image 103 into the noise removal / interpolation model 104, and outputs the noise removal / interpolation removed printout image 17DN (see reference) from the noise removal / interpolation model 104. Figure 18 The noise removal / interpolation unit 101 outputs the noise-removed / interpolated printed image 17DN to the authentication unit 50.

[0117] As an example, such as Figure 18 As shown, the marked image 16Y and the image 17DN of the printed copy after noise removal / interpolation are input to the authentication unit 50. Furthermore, in addition to the first setting range 38, the second setting range 105 is also input to the authentication unit 50. The second setting range 105 is stored in the storage device 20B.

[0118] As an example, the certification department passed 50. Figure 19 and Figure 20 The authentication is performed according to the processing steps shown in the flowchart ST320B. The processing of step ST320B is performed in place of the processing of step ST320A in the first embodiment described above. In addition, the same step numbers are added to the processing steps that are the same as those in the first embodiment described above.

[0119] First, similar to the first embodiment described above, the authentication unit 50 reads all dynamic image information 57 stored in the storage area 55 corresponding to the user ID of the retrieval request 80, and derives the feature values ​​of the ambient light removed printout image 17R and each of the marked images 16 and 16A to 16E of the marked image group 56 of the read dynamic image information 57 (step ST3201). Next, the authentication unit 50 derives the similarity between the ambient light removed printout image 17R and each of the marked images 16 and 16A to 16E (step ST3202).

[0120] The authentication unit 50 compares the exported similarity with the first set range 38 (step ST3203). If there is a marker image 16X whose similarity to the ambient light removed print image 17R is within the first set range 38 ("Yes" in step ST3203), the authentication unit 50 outputs the motion image 15 associated with the marker image 16X to the distribution control unit 48 (step ST3204).

[0121] On the other hand, if there is no marker image 16X whose similarity to the image 17R of the printed document after ambient light removal is within the first set range 38 (No in step ST3203), the authentication unit 50 compares the derived similarity with the second set range 105 (step ST3205). If there is a marker image 16Y whose similarity to the image 17R of the printed document after ambient light removal is within the second set range 105 (Yes in step ST3205), such as Figure 17 As shown, the differential extraction unit 100 extracts a differential image 103 between the marker image 16Y and the ambient light removed print image 17R (step ST3206), and outputs the differential image 103 to the noise removal / interpolation unit 101. The noise removal / interpolation unit 101 uses the noise removal / interpolation model 104 to remove noise from the ambient light removed print image 17R and performs interpolation, thereby converting the ambient light removed print image 17R into a noise-removed / interpolated print image 17DN (step ST3207). The noise removal / interpolation unit 101 outputs the noise-removed / interpolated print image 17DN to the authentication unit 50.

[0122] The authentication unit 50 exports the similarity between the noise-removed / interpolated printed image 17DN and the marker image 16Y (step ST3208). The authentication unit 50 compares the exported similarity with the first set range 38 (step ST3209). If the similarity between the noise-removed / interpolated printed image 17DN and the marker image 16Y is within the first set range 38 ("Yes" in step ST3209), the authentication unit 50 outputs the motion image 15 associated with the marker image 16Y to the distribution control unit 48 (step ST3204).

[0123] On the other hand, if there is no marker image 16Y whose similarity to the ambient light removed print image 17R is within the second set range 105 ("No" in step ST3205) and if the similarity between the noise removed / interpolated print image 17DN and the marker image 16Y is not within the first set range 38 ("No" in step ST3209), the authentication unit 50 will output the content of the motion image 15 that could not be retrieved to the distribution control unit 48.

[0124] Thus, in the second embodiment, the difference extraction unit 100 and the noise removal / interpolation unit 101 of the image quality correction processing unit 46 perform the following processing: removing and interpolating noise, such as the stamp object 102 (exemplified), that exists in the printout of the mark image 16 and becomes an obstacle when retrieving the mark image 16 corresponding to the printout captured image 17. Therefore, the possibility of false authentication caused by noise can be reduced.

[0125] Alternatively, the aforementioned differential extraction and noise removal / interpolation processing can be applied to the marked image 16X and the ambient light removed printed image 17R. In this case, if the similarity between the marked image 16X and the ambient light removed printed image 17R is lower than before the noise removal / interpolation processing, then the marked image 16X is used; conversely, if the similarity is higher, then the marked image 16X is not used.

[0126] [Third Implementation]

[0127] As an example, such as Figure 21 As shown, the third setting range 110 is input to the RW control unit 47 of the third embodiment. The third setting range 110 is stored in the storage device 20B. The third setting range 110 is an example of a "setting range" according to the technology of the present invention.

[0128] When a registration request 75 is input from the request receiving unit 45, the RW control unit 47 exhaustively compares and calculates the similarity between the tag image 16 to be registered contained in the registration request 75 and all registered tag images 16 in the storage area 55 of the dynamic image information DB36 corresponding to the user ID of the registration request 75. The RW control unit 47 compares the calculated similarity with a third setting range 110. If there is a registered tag image 16 whose similarity is within the third setting range 110, the RW control unit 47 uses the distribution control unit 48 to display a second notification screen 115 (see reference). Figure 22 The screen data is distributed to the user terminal 10 of the source of the registration request 75.

[0129] As an example, such as Figure 22 As shown, the browser control unit 32 receives the screen data of the second notification screen 115, reproduces the second notification screen 115 based on the screen data, and displays it on the monitor 24A. The second notification screen 115 displays a message 116 with the following content: a tag image 16 similar to the tag image 16 to be registered has been registered, and asks the user U whether to add an identifier 120 (see reference) to the tag image 16 to be registered. Figure 23 Then register. And, on the second notification screen 115, there are a yes button 117 and a no button 118.

[0130] When user U selects button 117, browser control unit 32 sends an identifier append request to dynamic image management server 14. Request receiving unit 45 receives the identifier append request and outputs it to RW control unit 47. As an example, such as Figure 23 As shown, the RW control unit 47 affixes an identifier 120, such as a QR (Quick Response) code (registered trademark), to each of the marker images 16 and 16A-16E constituting the marker image group 56. Then, the marker image group 56 with the affixed identifier 120 is associated with the moving image 15 and registered in the moving image information DB36. With the identifier 120 affixed in this way, the identifier 120 is also printed on the instant imaging film 12 along with the marker images 16. Furthermore, in Figure 23 The example shown is the case where identifier 120 is appended to the marked image 16.

[0131] On the other hand, if user U selects the No button 118, RW control unit 47 uses distribution control unit 48 to display the third notification screen 125 (see reference). Figure 24 The screen data is distributed to the user terminal 10 of the source of the registration request 75.

[0132] As an example, such as Figure 24As shown, the browser control unit 32 receives the screen data from the third notification screen 125, reproduces the third notification screen 125 based on the screen data, and displays it on the monitor 24A. The third notification screen 125 displays the content that the registered tag image 16 cannot be registered and a message 126 urging the user U to try registering other tag images 16. Furthermore, a confirmation button 127 is displayed on the third notification screen 125. When the confirmation button 127 is selected, the third notification screen 125 is closed.

[0133] As an example, Figure 25 and Figure 26 The flowchart shown illustrates the processing steps of the dynamic image management server 14 in the third embodiment. Furthermore, the same step numbers are added to the processing steps that are the same as those in the first embodiment described above.

[0134] After the registration request 75 is received by the request receiving unit 45 (Yes in step ST200), the similarity between the marker image 16 to be registered contained in the registration request 75 and all marker images 16 already registered in the storage area 55 of the dynamic image information DB36 corresponding to the user ID of the registration request 75 is derived in the RW control unit 47 (step ST230).

[0135] In the RW control unit 47, the derived similarity is compared with the third setting range 110 (step ST240). If there is no registered marker image 16 whose similarity is within the third setting range 110 (No in step ST240), similarly to the first embodiment described above, the marker image 16 of the registration request 75 is processed according to the type of the live imaging film 12 and the fading processing is performed by the film type corresponding processing unit 60 and the fading processing unit 61 of the image quality correction processing unit 46 (step ST210). Then, under the control of the RW control unit 47, the moving image 15 and the marker image group 56, i.e., the moving image information 57, are registered in the storage area 55 corresponding to the user ID of the registration request 75 (step ST220).

[0136] On the other hand, if there is a registered tagged image 16 with a similarity within the third set range 110 ("Yes" in step ST240), under the control of the distribution control unit 48, Figure 22 The screen data of the second notification screen 115 shown is distributed to the user terminal 10 of the source of the registration request 75, and thus the second notification screen 115 is displayed on the display 24A of the user terminal 10.

[0137] If the "Yes" button 117 on the second notification screen 115 is selected and the request receiving unit 45 receives an identifier attachment request from the browser control unit 32 ("Yes" in step ST250), after the image quality correction processing unit 46 performs image quality correction processing (step ST210), as follows: Figure 23 As shown, the RW control unit 47 adds identifiers 120 to each of the marker images 16 and 16A to 16E constituting the marker image group 56 (step ST260). Then, under the control of the RW control unit 47, the motion image 15 and the marker image group 56, i.e., the motion image information 57, are registered in the storage area 55 corresponding to the user ID of the registration request 75 (step ST220).

[0138] If the "No" button 118 on the second notification screen 115 is selected ("No" in step ST250), under the control of the distribution control unit 48, Figure 24 The screen data of the third notification screen 125 shown is distributed to the user terminal 10, the source of the registration request 75, and thus the third notification screen 125 is displayed on the display 24A of the user terminal 10.

[0139] Thus, in the third embodiment, the RW control unit 47 calculates the similarity between the tag image 16 to be registered and the already registered tag images 16. If there is an already registered tag image 16 whose similarity to the tag image 16 to be registered is within the third set range 110, the RW control unit 47 does not register the tag image 16 to be registered. Alternatively, the RW control unit 47 attaches an identifier 120 to the tag image 16 to be registered and registers it; the identifier 120 is used to distinguish it from the already registered tag images 16. Therefore, the possibility of false authentication caused by similar tag images 16 can be reduced.

[0140] The request receiving unit 45 receives requests such as identifier attachment requests based on the user U's instruction regarding whether to attach identifier 120 to the marker image 16 to be registered and to register it. Therefore, the user U's intention can be reflected in the registration of the marker image 16.

[0141] Furthermore, if there is a registered tag image 16 whose similarity to the tag image 16 to be registered is within the third setting range 110, the tag image 16 to be registered may not be registered without waiting for the user U's selection. Similarly, if there is a registered tag image 16 whose similarity to the tag image 16 to be registered is within the third setting range 110, the tag image 16 to be registered may be affixed with an identifier 120 and registered without waiting for the user U's selection.

[0142] [Fourth Implementation]

[0143] As an example, such as Figure 27 As shown, the CPU 22B of the dynamic image management server 14 in the fourth embodiment, in addition to the processing units 45-48 and 50 of the first embodiment described above (in... Figure 27 Besides the authentication unit 50, which is only shown in the diagram, it also functions as a film type determination unit 130. A printout image 17 is input to the film type determination unit 130. The film type determination unit 130 extracts the print area 12A and the border area 12B from the printout image 17. The film type determination unit 130 determines the type of the original instant imaging film 12 of the printout image 17 based on the pixel values ​​of the border area 12B. Then, a determination result 131 containing the determined type is output to the authentication unit 50. The authentication unit 50 retrieves the marker image 16X based on the type corresponding to the determination result 131. For example, if the determination result 131 is monochrome instant imaging film 12, the authentication unit 50 retrieves the marker image 16X based on the marker image 16B corresponding to monochrome film. This reduces the time required to retrieve the marker image 16X.

[0144] Alternatively, the date and time of the photograph printed on the photographed image 17 can be used for character recognition, and the marker image 16X can be retrieved within the elapsed period corresponding to the date and time of the photographed image. For example, if the date and time of the photographed image are 3 years ago, the marker image 16X can be retrieved within the 3-year faded marker image 16D. This can also shorten the time required to retrieve the marker image 16X.

[0145] [Fifth Implementation]

[0146] The above embodiments illustrate a method in which one user U enjoys the video transmission service, but are not limited to this. For example, such as... Figure 28 As shown, multiple users U can enjoy the video transmission service. Here, two users, UA and UB, are examples of multiple users U. Users UA and UB are acquaintances such as relatives, friends, or colleagues. User UA transfers a video 15 he / she took to user UB. In addition, user UA also transfers an instant imaging film 12 (a printout of the marked image 16) that he / she printed using the instant camera 11 to user UB.

[0147] User UB imports the animated image 15 transferred from user UA into its own user terminal 10B. Furthermore, user UB uses the camera function of its own user terminal 10B to take a printout of the marker image 16 transferred from user UA in order to register the marker image 16. Hereinafter, the image obtained by taking the printout of marker image 16 for registration will be labeled marker image 16P. User UB uses browser control unit 32 to send a registration request 75 for the imported animated image 15 and the captured marker image 16P to animated image management server 14.

[0148] As an example, such as Figure 29 As shown, in the fifth embodiment, when the request receiving unit 45 receives the registration request 75, the ambient light removal unit 62 also performs processing in addition to the film type corresponding processing unit 60 and the fading processing unit 61. The ambient light removal unit 62 inputs the marking image 16P of the registration request 75 to the ambient light removal filter 70. Then, it outputs the ambient light removed marking image 16PR from the ambient light removal filter 70. More specifically, the ambient light removal unit 62 extracts the printing area 12A and the border area 12B from the marking image 16P. The ambient light removal unit 62 determines the type of ambient light based on the pixel value of the border area 12B. Then, it selects the ambient light removal filter 70 corresponding to the determined type and applies it to the marking image 16P. The ambient light removal filter 70 selectively performs ambient light removal processing on the printing area 12A. The ambient light removal unit 62 outputs the ambient light removed marking image 16PR to the film type corresponding processing unit 60. Since the following processing is the same as in the first embodiment described above, the description is omitted.

[0149] Furthermore, the symbols 16PRA and 16PRB in the marked image group 56 represent the color film corresponding marked image and the monochrome film corresponding marked image, respectively, output by applying a color film 3D-LUT 65 and a monochrome film 3D-LUT 66 to the marked image 16PR after ambient light removal in the film type corresponding processing unit 60. Additionally, the symbols 16PRC, 16PRD, and 16PRE represent the 1-year fading corresponding marked image, the 3-year fading corresponding marked image, and the 5-year fading corresponding marked image, respectively, output by applying a 1-year fading correction 3D-LUT 67, a 3-year fading correction 3D-LUT 68, and a 5-year fading correction 3D-LUT 69 to the color film corresponding marked image 16PRA in the fading processing unit 61.

[0150] Thus, in the fifth embodiment, a marker image 16 is registered and printed on the instant imaging film 12 exchanged between users U. Therefore, multiple users U can enjoy the motion picture transmission service.

[0151] Furthermore, the ambient light removal unit 62 performs processing to remove the influence of ambient light from the image obtained from the printout of the marker image 16, i.e., the marker image 16P. Therefore, the possibility of false authentication due to ambient light can be reduced.

[0152] Furthermore, the image quality correction processing unit 46 only needs to have at least one of the illustrated film type corresponding processing unit 60, fading processing unit 61, and ambient light removal unit 62. For example, the ambient light removal unit 62 can be omitted, and the image quality correction processing unit 46 can be composed of the film type corresponding processing unit 60 and the fading processing unit 61. In this case, image quality correction processing is performed only on the marked image 16, and not on the printed image 17.

[0153] Alternatively, if the moving image 15 is stored in the storage device 20A of the user terminal 10, a display indication signal for the replay of the moving image 15 associated with the retrieved marker image 16X may be output instead of the moving image 15 associated with the retrieved marker image 16X.

[0154] Examples of processing corresponding to the type of instant imaging film 12 include processing for color instant imaging film 12 and processing for monochrome instant imaging film 12, but the examples are not limited to these. It is also possible to use only either the processing for color instant imaging film 12 or the processing for monochrome instant imaging film 12.

[0155] The printing medium is not limited to the illustrated instant imaging film 12. Photographic paper such as L-size paper can also be used. The captured image is not limited to the captured image 17 obtained by capturing a printout from the illustrated instant imaging film 12. It can also be a captured image of a marker image 16 displayed on a display device such as the display 24A of the user terminal 10.

[0156] The animated image 15 is used as an example of content, but it is not limited to this. It can also be 3D computer graphics of products such as cars, furniture, and clothing. Furthermore, it can be 3D computer graphics for tourist guides created by autonomous systems.

[0157] The hardware structure of the computer constituting the motion image management server 14 can be modified in various ways. For example, to improve processing power and reliability, the motion image management server 14 can be composed of multiple computers that are separate hardware components. For instance, the functions of the request receiving unit 45 and the image quality correction processing unit 46, and the functions of the RW control unit 47 and the distribution control unit 48 can be distributed among two computers. In this case, the motion image management server 14 is composed of two computers. Furthermore, the user terminal 10 can also perform all or part of the functions of the motion image management server 14.

[0158] Thus, the hardware structure of the user terminal 10 and the dynamic image management server 14 can be appropriately modified according to the performance requirements such as processing power, security, and reliability. Moreover, not limited to hardware, the dynamic image AP 30 and the operating program 35, etc., can also be dual-configured or distributed across multiple storage devices for the purpose of ensuring security and reliability.

[0159] In the above embodiments, for example, as the hardware structure of the processing units that perform various processes, such as the browser control unit 32, request receiving unit 45, image quality correction processing unit 46, RW control unit 47, distribution control unit 48, authentication unit 50, film type correspondence processing unit 60, fading processing unit 61, ambient light removal unit 62, difference extraction unit 100, noise removal / interpolation unit 101, and film type determination unit 130, various processors as shown below can be used. Among the various processors, in addition to general-purpose processors such as CPU 22A and 22B that execute software (moving image AP30 and working program 35) and function as various processing units, processors with circuit structures specifically designed for performing specific processes, such as FPGA (Field Programmable Gate Array), which can have their circuit structure changed after manufacturing, i.e., Programmable Logic Device (PLD), and / or ASIC (Application Specific Integrated Circuit), i.e., dedicated circuits, are also included.

[0160] A processing unit can consist of one of these various processors, or it can consist of a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Furthermore, multiple processing units can also be composed of a single processor.

[0161] As examples of multiple processing units comprised of a single processor, firstly, in computers such as client and server computers, a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Secondly, in systems-on-a-chip (SoC), a processor that implements the overall functionality of a system containing multiple processing units is used through a single integrated circuit (IC) chip. Thus, various processing units are constructed using one or more of the aforementioned processors as their hardware structure.

[0162] Furthermore, as the hardware structure of these various processors, more specifically, circuits composed of circuit elements such as semiconductor elements can be used.

[0163] Based on the above records, one can master the techniques described in the following notes.

[0164] [Note 1]

[0165] An image processing apparatus comprising a processor, The processor performs the following processing: Acquire captured images, said captured images being obtained by capturing marked images that are associated with content and become the output opportunity of the content; and Image quality correction is applied to at least one of the marked image or the captured image to reduce the image quality difference between the marked image and the captured image.

[0166] [Note 2]

[0167] According to the image processing apparatus described in Appendix 1, wherein, The processor retrieves the marker image corresponding to the captured image that has undergone the image quality correction processing, the marker image corresponding to the captured image that has undergone the image quality correction processing, or the marker image corresponding to the captured image that has undergone the image quality correction processing.

[0168] [Note 3]

[0169] According to the image processing apparatus described in Appendix 2, wherein, The processor outputs the content associated with the retrieved tagged image.

[0170] [Note 4]

[0171] The image processing apparatus according to any one of appendices 1 to 3, wherein, The captured image is a printed image obtained by photographing a printout of the marked image. As part of the aforementioned image quality correction process. The processor performs at least one of the following processes: The processing is performed in accordance with the type of printing medium used for printing the marked image; Perform processing corresponding to the fading of the printed copy of the marked image over the years; Remove the influence of ambient light when the printed copy of the marked image is captured; and The process involves removing noise from the printed copy of the marked image that would interfere with the retrieval of the marked image corresponding to the captured image, and then performing interpolation.

[0172] [Note 5]

[0173] The image processing apparatus according to any one of appendices 1 to 4, wherein, The processor associates and registers the content and the marked image.

[0174] [Note 6]

[0175] According to the image processing apparatus described in Appendix 5, wherein, The processor performs the following processing: Export the similarity between the tagged image to be registered and the already registered tagged images; and In the case where there is a registered tag image whose similarity to the tag image to be registered is within a set range, The tag image to be registered may be either not registered, or an identifier may be appended to the tag image to be registered and then registered, the identifier being used to distinguish it from the already registered tag image.

[0176] [Note 7]

[0177] According to the image processing apparatus described in Appendix 6, wherein, The processor receives instructions from the user regarding whether to attach an identifier to the tagged image to be registered and perform the registration.

[0178] [Note 8]

[0179] The image processing apparatus according to any one of appendices 1 to 7, wherein, The captured image is a photographic image of a printed copy of the marked image, and the printing medium for the printed copy of the marked image is instant imaging film.

[0180] The technology of the present invention can also be appropriately combined with the various embodiments and / or variations described above. Furthermore, it is not limited to the embodiments described above; various structures can be adopted as long as they do not depart from the spirit of the invention. Moreover, in addition to programs, the technology of the present invention also relates to storage media for non-transitory storage of programs and computer program products containing programs.

[0181] The above descriptions and illustrations are detailed explanations of the parts involved in the technology of this invention, and are merely one example of the technology of this invention. For example, the descriptions related to the above structure, function, effect, and effect are only one example of the structure, function, effect, and effect of the parts involved in the technology of this invention. Therefore, without departing from the spirit of the technology of this invention, unnecessary parts can be deleted, new elements can be added, or replacements can be made to the above descriptions and illustrations. Furthermore, to avoid cumbersome descriptions and to facilitate understanding of the parts involved in the technology of this invention, relevant explanations such as common technical knowledge that do not require special explanation in enabling the implementation of the technology of this invention have been omitted from the above descriptions and illustrations.

[0182] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, when "and / or" is used in this specification to represent three or more items together, the same concept as "A and / or B" can also be applied.

[0183] All documents, patent applications and technical standards described in this specification are incorporated herein by reference to the same extent as the specific documents, patent applications and technical standards described therein.

Claims

1. An image processing apparatus comprising a processor, The processor performs the following processing: Acquire captured images, said captured images being obtained by capturing marked images associated with the content and serving as output opportunities for said content; and Image quality correction processing is performed on at least one of the marked image and the captured image to reduce the image quality difference between the marked image and the captured image.

2. The image processing apparatus according to claim 1, wherein, The processor retrieves the marker image corresponding to the captured image that has undergone the image quality correction process, or the marker image corresponding to the captured image that has undergone the image quality correction process, or the marker image corresponding to the captured image that has undergone the image quality correction process.

3. The image processing apparatus according to claim 2, wherein, The processor outputs the content associated with the retrieved tagged image.

4. The image processing apparatus according to claim 1, wherein, The captured image is a printed image obtained by photographing a printout of the marked image. The processor performs at least one of the following processes as the image quality correction process: Perform processing corresponding to the type of printing medium used for the printout of the marked image; Perform processing corresponding to the fading of the printed copy of the marked image over the years; Remove the influence of ambient light when photographing the printed copy of the marked image; and Noise present in the printout of the marked image that would interfere when retrieving the marked image corresponding to the captured image is removed, and interpolation is performed.

5. The image processing apparatus according to claim 1, wherein, The processor registers the content in association with the tagged image.

6. The image processing apparatus according to claim 5, wherein, The processor performs the following processing: Export the similarity between the tagged image to be registered and the already registered tagged images; and In the case where there is a registered tagged image whose similarity to the tagged image to be registered is within a set range, The tag image to be registered may be either not registered, or an identifier may be appended to the tag image to be registered and then registered, the identifier being used to distinguish it from the already registered tag image.

7. The image processing apparatus according to claim 6, wherein, The processor receives instructions from the user regarding whether to attach an identifier to the tagged image to be registered and register it.

8. The image processing apparatus according to claim 1, wherein, The captured image is a photographic image of a printed copy of the marked image, and the printing medium used for the printed copy of the marked image is instant imaging film.

9. A method of operating an image processing apparatus, comprising the following steps: Acquire captured images, said captured images being obtained by capturing marked images associated with the content and serving as output opportunities for said content; and Image quality correction processing is performed on at least one of the marked image and the captured image to reduce the image quality difference between the marked image and the captured image.

10. An image processing apparatus operating procedure that causes a computer to perform a process comprising the following steps: Acquire captured images, said captured images being obtained by capturing marked images associated with the content and serving as output opportunities for said content; and Image quality correction processing is performed on at least one of the marked image and the captured image to reduce the image quality difference between the marked image and the captured image.

Images