Printing system, information processing device, program, and information processing method

Abstract

The present invention provides a printing system, information processing device, method, and program for evaluating user interest in content images. [Solution] In a printing system process including user print recording processing, content print recording processing, and interest level processing, the interest level processing determines a first-type similarity, which represents the similarity between a first-type designated image and a specific content image, using a first-type vector representing the features of the first-type designated image and a specific vector representing the features of the specific content image. A second-type similarity, which represents the similarity between a second-type designated image and a specific content image, is determined using a second-type vector representing the features of the second-type designated image and a specific vector. Interest level information, which is determined using the first-type similarity and the second-type similarity, and indicates the level of user interest in the specific content image, is stored in a storage device.

Description

Technical Field

[0001] This specification relates to a technique for printing content images.

Background Art

[0002] Content images representing various contents such as photographs, documents, advertisements, etc. are printed. Patent Document 1 discloses a technique for sending advertisement data, which is an example of content, to a printing device. Specifically, in response to a printing start request from a printing device, an advertisement transmission server extracts personal information about the user who is the request source from a customer information database. Based on the extracted personal information, the advertisement transmission server extracts advertisement data associated with target audience information indicating that the user is included in a consumer layer from an advertisement database, and sends it to the printing device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The images printed by a user represent the objects of the user's interest. For example, a user interested in landscape photographs can print various landscape photographs. Here, when a content image representing a car is printed, by observing the printed content image, the user may become interested in cars. Thus, by printing a content image different from the image printed by the user, the user's interest can change. There was room for improvement in evaluating the user's interest in content images.

[0005] This specification discloses a technique for evaluating the user's interest in content images.

Means for Solving the Problems

[0006] The technologies disclosed herein can be implemented as follows:

[0007] [Item 1] A printing system comprising a printing execution unit and a controller system including one or more controllers, wherein the controller system performs a first printing process to cause the printing execution unit to print a first type designated image which is an image specified by a user, and after the first printing process, performs a content printing process to cause the printing execution unit to print a specific content image which is different from the first type designated image, and after the content printing process, performs a second printing process to cause the printing execution unit to print a second type designated image which is an image which is different from the specific content image and is an image specified by the user, and the first type designated image and the specific content A printing system that performs a first determination process to determine a first type similarity representing the similarity between an image and a specific content image using a first type vector representing the features of the first type designated image and a specific vector representing the features of the specific content image; performs a second determination process to determine a second type similarity representing the similarity between the second type designated image and the specific content image using a second type vector representing the features of the second type designated image and the specific vector; and performs a storage process to store interest level information, which is determined using the first type similarity and the second type similarity, and indicates the level of interest of the user to the specific content image, in a storage device.

[0008] According to this configuration, interest level information indicating the user's level of interest in the specific content image is stored in the storage device using the first-type similarity between the first-type designated image and the specific content image, and the second-type similarity between the second-type designated image and the specific content image. Therefore, by referring to the interest level information, it is possible to evaluate the user's interest in the specific content image.

[0009] [Item 2] Information processing apparatus comprising: a first determination processing unit that performs a first determination process to determine a first type similarity, which represents the similarity between a first type designated image, which is an image specified by a user, and a specific content image different from the first type designated image, using a first type vector representing the features of the first type designated image and a specific vector representing the features of the specific content image; a second determination processing unit that performs a second determination process to determine a second type similarity, which represents the similarity between a second type designated image, which is an image different from the specific content image and specified by the user, and the specific content image, using a second type vector representing the features of the second type designated image and the specific vector; and a storage processing unit that performs a storage process to store interest level information, which is determined using the first type similarity and the second type similarity, and which indicates the level of interest of the user to the specific content image, in a storage device.

[0010] According to this configuration, interest level information indicating the user's level of interest in the specific content image is stored in the storage device using the first-type similarity between the first-type designated image and the specific content image, and the second-type similarity between the second-type designated image and the specific content image. Therefore, by referring to the interest level information, it is possible to evaluate the user's interest in the specific content image.

[0011] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, as information processing methods, information processing devices, servers for information processing, printing methods, printing systems, printers, printer controller systems, computer programs for implementing the functions of such methods or devices, and recording media (e.g., non-temporary recording media) on which such computer programs are recorded. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram illustrating an example of a printing system. [Figure 2] This is a sequence diagram illustrating an example of a printing system process. [Figure 3] This is a sequence diagram illustrating an example of a printing system process. [Figure 4] This is a sequence diagram illustrating an example of a printing system process. [Figure 5] This is a flowchart illustrating an example of the process for generating embedded vectors. [Figure 6] (A), (B), and (C) are block diagrams representing the generation of embedding vectors. (D) is a block diagram representing the calculation of the first kind vector V1. (E) is a block diagram representing the calculation of similarity. [Figure 7] This is a diagram illustrating an example of the record database 420. [Figure 8] This figure illustrates another example of content image distribution and printing. [Figure 9] This diagram shows an example of the information recorded in the record database 420. [Figure 10] This is a block diagram that shows how similarity is calculated. [Figure 11] This is a diagram illustrating another example of a record database. [Figure 12] This is a diagram illustrating another example of a record database. [Modes for carrying out the invention]

[0013] A. First Example: A1. System configuration: Figure 1 is a block diagram representing an embodiment of a printing system. This printing system 1000 includes a printer 100 and a service server 300. These devices 100 and 300 are connected to a network IT. The network IT may include the so-called Internet. The network IT may also include a so-called local area network.

[0014] The printer 100 includes a processor 110, a storage device 115, a display unit 140, an operation unit 150, a printing execution unit 160, a reading execution unit 170, and a communication interface 180. These elements are connected to each other via a bus. The storage device 115 includes a volatile storage device 120 and a non-volatile storage device 130.

[0015] The display unit 140 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 150 is a device configured to receive operations by a user, such as buttons, levers, or a touch panel disposed over the display unit 140. The display unit 140 and the operation unit 150 may form a so-called touch screen. The user can input various requests and instructions to the printer 100 by operating the operation unit 150. The display unit 140 may display operation elements such as buttons and sliders, and the displayed elements may be operated through the operation of the operation unit 150.

[0016] The printing execution unit 160 is a device that prints an image on a sheet such as paper. In this embodiment, the printer 100 is a so-called inkjet printer. The printing execution unit 160 includes a head having a plurality of nozzles, a drive circuit for driving the head, a plurality of conveyance rollers for conveying the sheet, and a motor for driving the plurality of conveyance rollers. The printing execution unit 160 is configured to eject one or more types of coloring materials from the plurality of nozzles to print an image on the sheet. The printing execution unit 160 of this embodiment prints an image using four colors of ink: cyan, magenta, yellow, and black. Note that the printer 100 may be a device that prints an image by another method, such as a laser printer.

[0017] The reading execution unit 170 is a device that optically reads an object such as a document. The reading execution unit 170 includes an optical sensor (not shown). The reading execution unit 170 generates scan data of a scan image representing the read object by optically reading the object. A printer including the reading execution unit is also called a multifunction device.

[0018] The communication interface 180 is an interface for communicating with other devices and includes, for example, one or more of a USB interface, a wired LAN interface, and a wireless interface of IEEE802.11. In this embodiment, the communication interface 180 is connected to the network IT.

[0019] The processor 110 is a device configured to perform data processing, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile memory device 120 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile memory device 130 is, for example, a flash memory. The non-volatile memory device 130 stores the data of the program PG1, the printer identifier IDd1, and the user information IDus, respectively.

[0020] The program PG1 is a program for controlling the printer 100. The data of the program PG1 is stored in the non-volatile memory device 130 by the manufacturer of the printer 100 during the manufacture of the printer 100. Alternatively, the program PG1 may be downloaded from a server not shown.

[0021] The printer identifier IDd1 is an identifier for identifying the printer 100. The user information IDus represents a user identifier for identifying a user. In the example of FIG. 1, the user information IDus represents a plurality of user identifiers IDu1, IDu2. The printer identifier IDd1 and the user identifiers IDu1, IDu2 are assigned to the printer 100 and the user, respectively, by the contract process of the distribution printing service.

[0022] In this embodiment, the service server 300 in Figure 1 is a cloud server using a cloud service. That is, the service server 300 is a virtual server provided by a cloud service provider that manages physical servers. The service provider starts operating the service server 300 by uploading the program PG3, which will be described later, to the cloud.

[0023] The service server 300 includes a processor 310, a storage device 315, and a communication interface 380. These elements are connected to each other via a bus (not shown). The storage device 315 includes a volatile storage device 320 and a non-volatile storage device 330.

[0024] The processor 310 is a device configured to perform data processing, such as a CPU or SoC. The volatile storage device 320 is such as DRAM, and the non-volatile storage device 330 is such as flash memory or a hard disk drive. The communication interface 380 is an interface for communicating with other devices, such as one or more of a USB interface, a wired LAN interface, or an IEEE 802.11 wireless interface. The communication interface 380 is connected to the network IT.

[0025] The services provided by the service server 300 may be various types of services. In this embodiment, the service server 300 provides a distribution and printing service. Specifically, the service server 300 causes a device such as a printer 100 to print a content image representing the content via the network IT. The content may be various types of content. For example, the content may be a landscape photograph, an animal photograph, a painting, or an advertisement. The printer 100 is an example of a device used for the service. Users can receive the service by registering their device with the service server 300. Although not shown in the diagram, the service server 300 provides the distribution and printing service to multiple printers.

[0026] The non-volatile storage device 330 stores data for program PG3, service database 410, record database 420, image embedding model MEi, text extraction model Mtxt, and text embedding model MEt. Program PG3 is configured to perform processing for the service. The service database 410 stores information used for the service. The service database 410 stores information for each delivery print service contract. The record database 420 stores information related to printed images. Image embedding model MEi is a machine learning model trained to generate embedding vectors, which are vectors representing image features, using images. Text extraction model Mtxt is a machine learning model trained to extract text from images. Text embedding model MEt is a machine learning model trained to generate embedding vectors, which are vectors representing text features, using text. Program PG3 and models MEi, Mtxt, and MEt are uploaded to the service server 300 by the service provider providing the service. The service database 410 is updated by the contract processing for the delivery print service. The record database 420 is updated by the print record processing described later. The details of the service database 410 are described below. Details of the record database 420 and models MEi, Mtxt, and MEt will be described later.

[0027] In this embodiment, the service database 410 stores the correspondence between a user identifier IDu that identifies a user, a printer identifier IDd that identifies a printer, and a delivery date and time DT. The delivery date and time DT is set to, for example, a date and time that repeats periodically. A date and time that repeats periodically includes, for example, the same time every day or the same time on the same day of the week every week.

[0028] To use the print distribution service, a user accesses the service server 300 using a terminal device (e.g., a smartphone or personal computer) not shown in the diagram, and applies to use the print distribution service. In response to the application, the terminal device and the service server 300 perform contract processing for the print distribution service. During contract processing, the processor 310 of the service server 300 records the correspondence between the user identifier IDu, the printer identifier IDd, and the distribution date and time DT in the service database 410. The user identifier IDu and the printer identifier IDd are assigned to the user and the printer, respectively, by the processor 310 of the service server 300. The distribution date and time DT is determined by the user. Hereinafter, a user of printer 100 will contract for the print distribution service that uses printer 100.

[0029] A terminal device (not shown) used for contract processing supplies data to the printer 100, consisting of a printer identifier assigned by the service server 300, in this case, printer identifier IDd1, and a user identifier, in this case, user identifier IDu1. The printer 100's processor 110 stores the received data in a non-volatile storage device 130. In the example shown in Figure 1, multiple users use the printer 100. Therefore, multiple user identifiers, IDu1 and IDu2, are associated with the same printer identifier IDd1.

[0030] The printer identifier IDd1 data may be stored in the non-volatile storage device 130 by the printer manufacturer at the time of printer 100's manufacture. This printer identifier IDd1 may then be used for the print distribution service.

[0031] Furthermore, the printer 100's processor 110 may manage the use of the printer 100 for each user. For example, a user may control the printer 100 by operating a terminal device such as a smartphone or personal computer that can communicate with the printer 100. Here, a user identifier may be associated with the terminal device. The printer 100's processor 110 may manage the use of the printer 100 for each user using the user identifier associated with the terminal device. For example, the processor 110 may count the number of pages printed for each user. As the user identifier associated with the terminal device, the user identifier used in the distribution printing service, for example, user identifiers IDu1, IDu2, may be used. Alternatively, the printer 100's processor 110 may assign a user identifier to each user. Such user identifiers may then be used in the distribution printing service.

[0032] A2. Printing system processing: Figures 2-4 are sequence diagrams illustrating an example of printing system processing performed by the printing system 1000. In this embodiment, the processor 110 of the printer 100 and the processor 310 of the service server 300 carry out the printing system processing. Hereinafter, the processor 110 of the printer 100 will also be referred to as the printer processor 110, and the processor 310 of the service server 300 will also be referred to as the server processor 310. The printer processor 110 carries out the printing system processing according to program PG1. The server processor 310 carries out the printing system processing according to program PG3.

[0033] The printing system processing includes user print recording processing (S110 in Figure 2 and S310 in Figure 3), content print recording processing (S210 in Figure 3), and interest level processing (S410 in Figure 4). Hereinafter, "user print recording processing" will also be referred to as "user processing," and content print recording processing will also be referred to as "content processing."

[0034] User processing S110 and S310 are processes that print a specified image designated by the user and record the embedding vector of the specified image. User processing S110 performed before content processing S210 is called first-type user processing S110. User processing S310 performed after content processing S210 is called second-type user processing S310. Content processing S210 is a process that prints the content image and records the embedding vector of the content image. Interest level processing S410 is a process that generates interest level information for the content image using the embedding vector and records the generated interest level information.

[0035] The user can freely print images of interest using the printer 100. The first type user processing S110 in Figure 2 proceeds when printing is performed at the user's instruction. The first type user processing S110 includes S120-S155. In S120, the user of the printer 100 logs in to the printer 100. In this embodiment, the user logs in to the printer 100 by connecting a terminal device (not shown) to the printer 100. This allows the user to use the printer 100. The connection between the terminal device and the printer 100 may be made via a network IT. The printer 100 may also be equipped with a wireless interface for short-range communication, such as a Bluetooth® interface or a Near Field Communication (NFC) interface. The terminal device and the printer 100 may be connected by short-range communication. Alternatively, the user may log in to the printer 100 by operating the operation unit 150 of the printer 100 without using a terminal device. Printing by printer 100 may be performed even if the user is not logged into printer 100. In other words, S120 may be omitted.

[0036] In S125, the printer processor 110, in accordance with the user's instructions, causes the print execution unit 160 to print a specified image designated by the user. The specified image may be any image. For example, the user may specify an image stored in a terminal device, such as a personal computer connected to the printer 100, by operating the terminal device. The terminal device sends the data of the specified image to the printer 100, and the printer processor 110 causes the print execution unit 160 to print the specified image. Alternatively, the user may input a copy instruction for an object, such as a document, to the printer 100. The printer processor 110 causes the read execution unit 170 to read the object, and the print execution unit 160 to print the scanned image. Hereinafter, the specified image used in S125 will also be referred to as the first type of specified image.

[0037] In S130, the printer processor 110 determines whether or not it can identify the user who issued the print command. In this embodiment, the printer processor 110 determines that it can identify the user when printing is performed while the user is logged into the printer 100.

[0038] If the result of the S130 decision is Yes, in S135 the printer processor 110 identifies the user who issued the print command and obtains the identifier of the identified user. In this embodiment, the printer processor 110 refers to the user information IDus in Figure 1 and obtains the user identifier of the logged-in user. Then the printer processor 110 proceeds to S140.

[0039] If the result of S130 is No, the printer processor 110 skips S135 and proceeds to S140.

[0040] In S140, the printer processor 110 sends print-related data to the service server 300. The print-related data includes data for the specified image, data for the printer identifier IDd1, and data for the print type. The print type indicates the type of print, such as copy or PC printing (printing from a personal computer). If the result of the S130 decision is Yes, the print-related data includes data for the user identifier.

[0041] In S145, the server processor 310 receives print-related data. In S150, the server processor 310 generates an embedded vector associated with the specified image. Hereinafter, the embedded vector generated in S150 will be referred to as a first-kind individual vector.

[0042] Figure 5 is a flowchart illustrating an example of the embedding vector generation process. The generated embedding vectors include image embedding vectors representing image features, text embedding vectors representing text features, and image-text embedding vectors representing both image and text features. In this embodiment, the type of embedding vector to be generated is predetermined. Alternatively, the type of embedding vector to be generated may be selected according to a rule. For example, the type of embedding vector may be selected according to the settings of the print distribution service.

[0043] Figure 5 shows an example flowchart that can generate any of the three types of embedded vectors. The generation process for each type of embedded vector will be explained below with reference to Figure 5. Note that if only a specific type of embedded vector is generated, the steps for generating the other types of embedded vectors may be omitted.

[0044] In S520, the server processor 310 determines whether or not text is considered in the generation of the embedded vector. The server processor 310 may make the determination in S520 by referring to, for example, the settings of the delivery print service. If text is not considered in the generation of the embedded vector (S520: No), the server processor 310 proceeds to S525.

[0045] In S525, the server processor 310 generates an image embedding vector using the specified image and the image embedding model MEi. Then, the server processor 310 completes the process shown in Figure 5, i.e., the process shown in S150 of Figure 2.

[0046] Figure 6(A) is a block diagram representing the generation of an embedding vector. The server processor 310 generates an embedding vector Vi of the specified image IM by performing calculations on the image embedding model MEi using the specified image IM. Hereinafter, the embedding vector Vi of the specified image IM will also be called the image embedding vector Vi. The dimensions of the embedding vector Vi may be various values. For example, the embedding vector Vi may be a 32-dimensional vector. The embedding vector Vi represents a point in the vector space SVi. Hereinafter, the model that generates an image embedding vector from an image will be called the vector generation model MVi. In this embodiment, the vector generation model MVi includes the image embedding model MEi.

[0047] The image embedding model MEi may be various models configured to generate vectors representing the features of an image. For example, a convolutional neural network containing one or more convolutional layers may be used as the image embedding model MEi. Examples of convolutional neural networks include VGG or ResNet. Here, the feature vectors output from the fully connected layers included in the model may be used as embedding vectors. The fully connected layer that outputs the embedding vectors may be the last fully connected layer of the model, or a fully connected layer upstream of the last fully connected layer.

[0048] Convolutional neural networks can classify multiple types of images based on their features. Feature vectors generated by convolutional neural networks can represent features associated with different image types. Such feature vectors can be used as embedding vectors representing image features. Pre-trained model data for various models such as VGG16, VGG19, ResNet18, and ResNet-50 are publicly available on the internet. These pre-trained models may be used as image embedding models (MEi).

[0049] If text is considered in the generation of the embedding vector (Figure 5: S520: Yes), in S530, the server processor 310 extracts text from the specified image. The image may represent various texts directed at the viewer of the image. For example, an advertising image may represent, for example, the name and description of the advertised item, such as a product or service. There may be various methods for extracting the text. In this embodiment, the server processor 310 uses the text extraction model Mtxt to extract text from the specified image. In S535, the server processor 310 uses the extracted text and the text embedding model MEt to generate a text embedding vector.

[0050] Figure 6(B) is a block diagram representing the generation of the embedding vector. The server processor 310 extracts the text TX represented by the specified image IM by performing operations on the text extraction model Mtxt using the specified image IM. The server processor 310 generates the embedding vector Vt of the text TX by performing operations on the text embedding model MEt using the extracted text TX. Hereinafter, the embedding vector Vt of the text TX will also be called the text embedding vector Vt. The dimensions of the embedding vector Vt may be various values. For example, the embedding vector Vt may be a 32-dimensional vector. The embedding vector Vt represents a single point in the vector space SVt. Hereinafter, the model that generates text embedding vectors from an image will be called the vector generation model MVt. In this embodiment, the vector generation model MVt includes the text extraction model Mtxt and the text embedding model MEt.

[0051] The text extraction model Mtxt may be any model configured to extract text from an image. For example, optical character recognition engines such as "Tesseract OCR" or "EasyOCR" may be used as the text extraction model Mtxt. Pre-trained model data for these engines is publicly available on the internet. Such pre-trained models may be used as the text extraction model Mtxt.

[0052] The text embedding model MEt may be any model configured to generate vectors representing the features of text. For example, a model called BERT (Bidirectional Encoder Representations from Transformers) may be used as the text embedding model MEt. The text to be processed is broken down into multiple words for input into the BERT model. Then, a sequence of tokens representing the sequence of words is input into the BERT model. A tokenizer is used to convert the text into a sequence of tokens. Pre-trained BERT model data and tokenizer data suitable for pre-trained BERT models are publicly available on the internet. Such pre-trained BERT models and tokenizers may be used as the text embedding model MEt.

[0053] After S535 in Figure 5, in S540, the server processor 310 determines whether or not to integrate the text embedding vector and the image embedding vector. The server processor 310 may make the decision in S540 by referring to, for example, the settings of the print distribution service. If integration is not required (S540: No), the server processor 310 terminates the process in Figure 5, i.e., the process in S150 in Figure 2.

[0054] If integration is required (S540: Yes), in S545, the server processor 310 generates an image embedding vector using the specified image and the image embedding model MEi. This process is performed in the same manner as in S525.

[0055] In S550, the server processor 310 generates a unified embedding vector by integrating the text embedding vector and the image embedding vector. Then, the server processor 310 completes the process shown in Figure 5, i.e., the process shown in S150 of Figure 2.

[0056] Figure 6(C) is a block diagram representing the generation of the embedding vector. As described in S530 and S535, the server processor 310 generates a text embedding vector Vt from the specified image IM using the specified image IM, the text extraction model Mtxt, and the text embedding model MEt. As described in S545, the server processor 310 generates an image embedding vector Vi from the specified image IM using the specified image IM and the image embedding model MEi. In S550, the server processor 310 generates a combined embedding vector Vit by integrating the embedding vectors Vi and Vt.

[0057] There are various methods for integrating the two vectors. For example, the server processor 310 may generate an integrated embedding vector Vit by combining the image embedding vector Vi and the text embedding vector Vt. In this case, the dimension of the integrated embedding vector Vit is the sum of the dimensions of the image embedding vector Vi and the dimensions of the text embedding vector Vt. Alternatively, if the dimension of the image embedding vector Vi is the same as the dimension of the text embedding vector Vt, the server processor 310 may set the integrated embedding vector Vit as the average vector of the image embedding vector Vi and the text embedding vector Vt. In either case, the integrated embedding vector Vit represents a single point in the vector space SVit. Hereinafter, the model that generates an integrated embedding vector from an image will be referred to as the vector generation model MVit. In this embodiment, the vector generation model MVit includes an image embedding model MEi, a text extraction model Mtxt, and a text embedding model MEt.

[0058] As described above, in S150 of Figure 2, the server processor 310 generates one of the following as an embedded vector associated with the specified image: an image embedded vector Vi, a text embedded vector Vt, or an integrated embedded vector Vit. The generated embedded vectors are embedded vectors associated with each individual specified image. Hereinafter, the embedded vectors generated in S150 will be referred to as Type 1 individual vectors.

[0059] In S155, the server processor 310 associates the first type individual vector, the printer identifier, the user identifier, and the time, and records them in the record database 420. Figure 7 shows an example of the record database 420. In this embodiment, the record database 420 represents print information PF, which is the correspondence between the print identifier IDp, the print type PT, the time T, the embedded vector V, the content identifier IDc, and the level of interest information D. One piece of print information PF represents one print. In S155, the server processor 310 records new print information PF in the record database 420.

[0060] The print identifier IDp is an identifier of the correspondence (i.e., print information PF). The server processor 310 sets the print identifier IDp to an identification number in ascending order, for example. The print type PT indicates the type of print. The print type is set to one of several types, including, for example, copy PTa, PC print PTb, and distribution print PTc. In S155, the print type PT is set to the print type included in the print-related data (S145). The time T is the time indicating the print. In S155, the time T is set to the current time. The current time is indicated by a clock (not shown) provided on the service server 300. The time T may represent, for example, year, month, day, hour, and minute. The embedded vector V is an embedded vector associated with the image to be printed. In S155, the embedded vector V is set to the first type individual vector generated in S150. The content identifier IDc is an identifier that identifies the content image to be printed by distribution print, which will be described later. The interest level information D is information that represents the user's level of interest in the content image. In S155, the setting of the content identifier IDc and interest level information D is omitted.

[0061] In this embodiment, the recording database 420 records print information PF for each combination of printer identifier IDd and user identifier IDu. In the example in Figure 7, the recording database 420 includes sub-databases 420a-420d, each containing a different combination of printer identifier IDd and user identifier IDu. In S155, the server processor 310 records new print information PF in the sub-database associated with the combination of printer identifier and user identifier included in the print-related data.

[0062] If the print-related data does not include a user identifier, the server processor 310 may identify the user identifier in various ways. For example, the server processor 310 may use the user identifier IDu, which is associated with the printer identifier IDd included in the print-related data by the service database 410 in Figure 1. If multiple user identifiers are associated with the printer identifier, the server processor 310 may use a representative user identifier that has been pre-selected from the multiple user identifiers. The representative user identifier may be selected, for example, during contract processing.

[0063] Upon completion of S155, the first type user processing S110 is terminated. The user can perform various print operations using the printer 100. That is, the first type user processing S110 may be performed multiple times before the content processing S210 in Figure 3. The three print information PFs represented by print identifiers IDp1-IDp3 in Figure 7 are recorded by three instances of the first type user processing S110. The embedded vectors Vi1-Vi3 associated with print identifiers IDp1-IDp3 are first type individual vectors. The user is free to print images of interest. For example, a user interested in cats may print various cat photographs.

[0064] Next, we will explain the content processing S210 in Figure 3. The server processor 310 starts printing the content image using the distribution printing service according to the distribution date and time DT in the service database 410 in Figure 1. In S220, the server processor 310 obtains data including the content image, a content identifier that identifies the content image, a user identifier, and a printer identifier.

[0065] There are various methods for obtaining content images. In this embodiment, the server processor 310 obtains the printer identifier IDd of the printer 100 to which the content image will be distributed and the user identifier IDu of the user from the service database 410 shown in Figure 1 for printing the content image. Next, the server processor 310 obtains the content image for the user of the user identifier IDu obtained from the service database 410. For example, the server processor 310 obtains the content image by auction. For example, the server processor 310 performs auction processing as a seller of advertising space and determines the successful bidder for the advertising space. The successful bidder's advertising image is used as the content image to be printed. The content identifier is pre-associated with the advertising image. For example, the successful bidder pre-associates the content identifier with the advertising image. Alternatively, the server processor 310 may assign the content identifier to the content image. There are various methods for conducting the auction. For example, the server processor 310 performs auction processing by communicating with a bidding server (not shown). Communication between the service server 300 and the bidding server may be performed according to a protocol called OpenRTB. RTB stands for "Real Time Bidding".

[0066] Furthermore, the server processor 310 obtains a user identifier from the service database 410 shown in Figure 1 that is associated with printing the content image in question.

[0067] In S225 of Figure 3, the server processor 310 generates data for the print image PI. The print image PI represents the content image acquired in S220. The print image PI may further represent the username associated with the user identifier as the destination, so that the user associated with the user identifier can view the printed print image PI.

[0068] The data format of the print image PI is a printer-independent format such as JPEG or PNG. Alternatively, the data format of the print image PI may be a format that can be associated with a printer. In either case, the color space of the print image PI data may be various color spaces. Examples of color spaces include the RGB color space, YCbCr color space, CMYK color space, and other color spaces usable by the printer.

[0069] In S230, the server processor 310 sends a print command to the printer 100. The destination printer is the one indicated by the printer identifier IDd, which is associated with printing the content image in this case by the service database 410 in Figure 1. The print command includes the data of the print image PI and the data of the content identifier. The print command is also called a print job. Note that the print command to the printer 100 may be sent via a remote print server (not shown).

[0070] In S250, the server processor 310 generates an embedded vector associated with the content image. The method for generating the embedded vector is the same as the method used in S150 in Figure 2, except that the content image is used instead of the Type 1 designated image. Hereafter, the embedded vector generated in S250 will be referred to as the specific vector.

[0071] In S255, the server processor 310 associates a specific vector, a printer identifier IDd, a content identifier IDc, a user identifier IDu, and a time T, and records them in the recording database 420 shown in Figure 7. Specifically, the server processor 310 selects a sub-database of the printer identifier IDd and user identifier IDu obtained in S220 from the recording database 420 shown in Figure 7, and associates the specific vector, content identifier IDc, and time T with the selected sub-database to record it as print information PF. The print information PF shown by the print identifier IDpc in Figure 7 is an example of print information PF recorded in S255. In S255, the print type PT is set to distribution print PTc, the time T is set to the current time Tc, the embedded vector V is set to the specific vector Vc generated in S250, and the content identifier IDc is set to the content identifier IDc1 obtained in S220. The interest level information D is not set in S255. The level of interest information D is set in the level of interest processing S410, which will be described later. The printer identifier IDd and user identifier IDu are the information associated with printing the content image in this case by the service database 410 (Figure 1).

[0072] Meanwhile, the printer 100 receives a print command in S260. In S265, the printer processor 110 prints the print image PI according to the print command. The user observes the printed print image PI. The user's interest in the print image PI may be high or low.

[0073] In S270, the printer processor 110 sends a print completion notification to the service server 300. This notification includes data for the content identifier. In S275, the server processor 310 receives the print completion notification. The server processor 310 refers to the content identifier included in the notification and confirms that the printing of the content image is complete. With this, content processing S210 is completed. The server processor 310 may execute S255 in response to receiving the print completion notification (S275).

[0074] After content processing S210, the user can freely print images of interest. Second-type user processing S310 in Figure 3 proceeds when printing is performed at the user's instruction. Although detailed illustration is omitted, second-type user processing S310 is the same process as first-type user processing S110 in Figure 2. Second-type user processing S310 includes several steps corresponding to S120-S155 of first-type user processing S110. S325, S350, and S355 are shown in the figure. S325 corresponds to S125 in Figure 2. In S325, the printer processor 110 causes the print execution unit 160 to print the specified image specified by the user, according to the user's instructions. The specified image specified by the user in S325 is called a second-type specified image. S350 corresponds to S150 in Figure 2. In S350, the server processor 310 generates embedded vectors associated with the Type 2 designated image using the same generation method as in S150 in Figure 2. The embedded vectors generated in S350 are called Type 2 individual vectors. S355 corresponds to S155 in Figure 2. In S355, the server processor 310 records the Type 2 individual vectors, printer identifiers, user identifiers, and time in association with each other in the recording database 420 using the same processing as in S155. The three print information PFs represented by print identifiers IDp4-IDp6 in Figure 7 are recorded by three Type 2 user processing S310s. The embedded vectors Vi4-Vi6 associated with print identifiers IDp4-IDp6 are Type 2 individual vectors.

[0075] A user's interests can vary. For example, the content image printed in S210 of Figure 3 may include a photograph of a car. By observing the printed content image, the user's interest in cars may increase. In the second type of user processing S310, following the content processing S210, a user interested in cars may print various photographs of cars. If the user's interest in the content image is low, the user may, after printing the content image, print a second type of designated image similar to the first type of designated image printed before the content image was printed.

[0076] After multiple Type 2 user processing S310, the server processor 310 starts the interest level processing S410 shown in Figure 4. In the interest level processing S410, the server processor 310 uses multiple Type 1 individual vectors generated in S150 of Figure 2, a specific vector generated in S250 of Figure 3, and multiple Type 2 individual vectors generated in S350 of Figure 3 to generate interest level information D for the content image shown in Figure 7. The server processor 310 may start the interest level processing S410 when a start condition is met that indicates that multiple Type 1 individual vectors and multiple Type 2 individual vectors are available. The start condition may be, for example, that N or more Type 1 individual vectors and N or more Type 2 individual vectors are available. N is an integer of 2 or more and is a predetermined lower limit for generating interest level information D. Alternatively, the start condition may be that a predetermined time has elapsed after the completion of content processing S210. Furthermore, the server processor 310 executes interest level processing S410 for each combination of printer identifier IDd and user identifier IDu.

[0077] The interest level processing S410 includes S420-S475. In S420, the server processor 310 selects the sub-database from the recording database 420 in Figure 7 that is the target of the interest level processing S410. Next, the server processor 310 refers to a plurality of individual vectors of type I recorded in the selected sub-database. Here, it is assumed that sub-database 420a is selected and that three individual vectors of type I Vi1-Vi3, associated with print identifiers IDp1-IDp3, are available.

[0078] In S425, the server processor 310 generates a vector V1 of the first kind using multiple individual vectors Vi1-Vi3. Figure 6(D) is a block diagram representing the calculation of the vector V1 of the first kind. The figure shows the designated images IM1-IM3 of the first kind, the vector generation model MV, the individual vectors Vi1-Vi3 of the first kind, and the vector V1 of the first kind. The vector generation model MV is the model used to generate the individual vectors Vi1-Vi3 of the first kind. The vector generation model MV is one of the vector generation models MVi, MVt, or MVit shown in Figures 6(A)-6(C). The individual vectors Vi1-Vi3 of the first kind each represent a point in the vector space SV. The server processor 310 determines the vector V1 of the first kind to be a vector that represents the multiple individual vectors Vi1-Vi3 of the first kind. In this embodiment, the server processor 310 determines the first kind vector V1 to be a vector representing the centroid of a plurality of individual first kind vectors Vi1-Vi3 in the vector space SV.

[0079] If any one of the multiple individual vectors of type I, Vi1-Vi3, changes, then the vector V1 of type I may also change. In this way, the vector V1 of type I can change according to each of the multiple individual vectors of type I, Vi1-Vi3, that is, according to each of the multiple designated images of type I, IM1-IM3. It can be said that such a vector V1 of type I represents the characteristics of each of the multiple designated images of type I, IM1-IM3.

[0080] In S430 of Figure 4, the server processor 310 stores the data of the first kind vector V1 in the storage device 315 (for example, the non-volatile storage device 330).

[0081] In S440, the server processor 310 references multiple individual vectors of type II in the record database 420 shown in Figure 7. Here, it is assumed that three individual vectors of type II, Vi4-Vi6, associated with print identifiers IDp4-IDp6, are available.

[0082] In S445, the server processor 310 generates a vector V2 of the second kind using multiple individual vectors Vi4-Vi6. The method for generating the vector V2 of the second kind is the same as the method for generating the vector V1 of the first kind in S425. The server processor 310 determines the vector V2 of the second kind to be a vector representing the centroid of multiple individual vectors Vi4-Vi6 in the vector space SV.

[0083] In S450, the server processor 310 stores the data of the second kind vector V2 in a storage device 315, for example, a non-volatile storage device 330.

[0084] In S460, the server processor 310 calculates the first-kind similarity SM1, which is the similarity between the first-kind vector V1 and a specific vector Vc. Figure 6(E) is a block diagram representing the calculation of the similarity. The figure shows the first-kind designated images IM1-IM3, the content image IMc, the second-kind designated images IM4-IM6, the vector generation model MV, the individual vectors Vi1-Vi6, the embedded vector Vc, the first-kind vector V1, and the second-kind vector V2.

[0085] The first-kind similarity SM1 may be various values ​​representing the similarity between the first-kind vector V1 and the embedded vector Vc. In this embodiment, the server processor 310 calculates the cosine similarity between the first-kind vector V1 and the embedded vector Vc as the first-kind similarity SM1. The first-kind similarity SM1 indicates the similarity between the content image IMc and a plurality of first-kind designated images IM1-IM3 that were printed before the content image IMc was printed.

[0086] In step S465 of Figure 4, the server processor 310 calculates the second-type similarity SM2, which is the similarity between the second-type vector V2 and a specific vector Vc. The method for calculating the second-type similarity SM2 is the same as the method for calculating the first-type similarity SM1 in step S460. The server processor 310 calculates the cosine similarity between the second-type vector V2 and the embedded vector Vc as the second-type similarity SM2. The second-type similarity SM2 represents the similarity between the content image IMc and the multiple second-type designated images IM4-IM6 that are printed after the content image IMc is printed.

[0087] In S470 of Figure 4, the server processor 310 determines interest level information D using similarity scores SM1 and SM2. Interest level information D represents the level of user interest in the content image IMc. If the user has a high level of interest in the content image IMc, they may print multiple images similar to the content image IMc after printing the content image IMc. In this case, the second-class similarity score SM2 may be higher than the first-class similarity score SM1. That is, a second-class similarity score SM2 higher than the first-class similarity score SM1 indicates a high level of user interest in the content image IMc. If the user has little interest in the content image IMc, they may print second-class designated images IM4-IM6 similar to the first-class designated images IM1-IM3 that were printed before printing the content image IMc. In this case, the second-class similarity score SM2 may be about the same as the first-class similarity score SM1. In other words, a Type 2 similarity score (SM2) that is similar to a Type 1 similarity score (SM1) indicates that users have little interest in the content image (IMc). Furthermore, if users have little interest in the content image (IMc), a Type 2 similarity score (SM2) may be lower than a Type 1 similarity score (SM1).

[0088] Thus, the level of user interest in the content image IMc can be represented by a combination of the first-type similarity SM1 and the second-type similarity SM2. The interest level information D determined in S470 may be various pieces of information that indicate the level of interest. For example, the interest level information D may be information Da representing the combination of similarity SM1 and SM2. Alternatively, the interest level information D may be information Db representing the difference obtained by subtracting the first-type similarity SM1 from the second-type similarity SM2. The larger the difference, the higher the user's interest. A positive difference indicates that the user is interested in the content image IMc. When the difference is a large positive value, the user's interest is particularly high. Furthermore, the interest level information D may be information Dc representing the sign of the difference obtained by subtracting the first-type similarity SM1 from the second-type similarity SM2. A positive sign indicates high interest, and a negative sign indicates low interest.

[0089] In S475, the server processor 310 associates the interest level information D, the content identifier IDc, the printer identifier IDd, and the user identifier IDu, and records them in the recording database 420. That is, in the sub-database 420a selected in S420, the server processor 310 records the interest level information D in association with the print identifier IDpc of the content image IMc used to calculate the similarity in this interest level processing. In the example in Figure 7, the interest level information D associated with the print identifier IDpc is set to the interest level information D1 determined in S470. With this, the interest level processing S410, and consequently the printing system processing in Figures 2-4, is completed.

[0090] After the completion of interest processing S410, the user can freely print images of interest using the printer 100. If printing is initiated by the user after the completion of interest processing S410, the first type user processing S110 shown in Figure 2 is executed. Then, a new printing system process proceeds. Note that the information recorded by the second type user processing S310 shown in Figure 3 may be used in place of the information recorded by the first type user processing S110 in the next printing system process.

[0091] As described above, in this embodiment, the printing system 1000 in Figure 1 comprises a print execution unit 160 and a controller system 1000c including a printer processor 110 and a server processor 310. Processors 110 and 310 are examples of controllers.

[0092] The controller system 1000c performs the following processes. In S125 of Figure 2, the controller system 1000c causes the print execution unit 160 to print the first type designated images IM1-IM3 shown in Figure 6(E). The process in S125 is an example of a first printing process in which the print execution unit 160 prints the first type designated images, which are images specified by the user. In S265 of Figure 3, following S125, the controller system 1000c causes the print execution unit 160 to print the content image IMc shown in Figure 6(E). The process in S265 is an example of a content printing process in which the print execution unit 160 prints a specific content image different from the first type designated images. After S265, the controller system 1000c performs the process in S310. S310 includes S325. In S325, the controller system 1000c causes the print execution unit 160 to print the second type designated images IM4-IM6 shown in Figure 6(E). The process in S325 is an example of a second printing process in which the print execution unit 160 prints a second-type designated image, which is an image different from the specific content image and is specified by the user.

[0093] In S460 of Figure 4, the controller system 1000c determines the first type similarity SM1 using the first type vector V1 and the specific vector Vc. The first type vector V1 represents the features of the first type designated images IM1-IM3. The specific vector Vc represents the features of the specific content image IMc. The first type similarity SM1 represents the similarity between the first type designated images IM1-IM3 and the specific content image IMc. The process in S460 is an example of a first determination process that determines the first type similarity between the first type designated images and the specific content images using the first type vector representing the features of the first type designated images and the specific vector representing the features of the specific content images. In S465, the controller system 1000c determines the second type similarity SM2 using the second type vector V2 and the specific vector Vc. The second type vector V2 represents the features of the second type designated images IM4-IM6. The second-order similarity SM2 represents the similarity between the second-order designated images IM4-IM6 and the specific content image IMc. The process in S465 is an example of a second determination process that determines the second-order similarity between the second-order designated images and the specific content image using a second-order vector representing the features of the second-order designated images and a specific vector. In S475, the controller system 1000c stores the level of interest information D, determined using the first-order similarity SM1 and the second-order similarity SM2, in the record database 420 shown in Figure 7. The level of interest information D indicates the level of user interest in the specific content image IMc. As shown in Figure 1, the record database 420 is stored in the storage device 315, in this case, the non-volatile storage device 330. Thus, in S475, the controller system 1000c stores the level of interest information D in the storage device 315. The process in S475 is an example of a storage process that stores the level of interest information in a storage device.

[0094] According to this configuration, interest level information D, which indicates the level of user interest in the specific content image IMc, is stored in the storage device 315 using the first-type similarity SM1 between the first-type designated images IM1-IM3 and the specific content image IMc, and the second-type similarity SM2 between the second-type designated images IM4-IM6 and the specific content image IMc. Therefore, the level of user interest in the specific content image IMc can be evaluated by referring to the interest level information D. The interest level information D may be referenced by various processes. For example, a person involved with the printing system 1000, such as a user of the printer 100, an employee of a distribution printing service provider, or an advertiser, may perform an evaluation process for the specific content image IMc. The evaluation process may include a process of evaluating the level of user interest by referring to the interest level information D in the record database 420 shown in Figure 7.

[0095] In this embodiment, the controller system 1000c determines the level of interest information D in S470 of Figure 4. The determined level of interest information D may be information Db representing the difference between similarity scores SM1 and SM2, or information Dc representing the sign of the difference (positive or negative). That is, in S470, the controller system 1000c may perform a level of interest determination process to determine the level of interest information D representing the difference between the second-type similarity score SM2 and the first-type similarity score SM1, or the sign of the difference (positive or negative). Such level of interest information D makes it easy to evaluate the level of interest of the user.

[0096] In this embodiment, the controller system 1000c executes the S125 process multiple times. The multiple S125 processes cause the print execution unit 160 to print the multiple Type 1 designated images IM1-IM3 shown in Figure 6(E). Thus, the first printing process, which causes the print execution unit 160 to print the Type 1 designated images, includes the process of printing multiple S125 processes, that is, the process of printing the multiple Type 1 designated images IM1-IM3, which are multiple images specified by the user, to the print execution unit 160.

[0097] The controller system 1000c executes the S325 process multiple times. The multiple S325 processes cause the print execution unit 160 to print the multiple Type 2 designated images IM4-IM6 shown in Figure 6(E). Thus, the second printing process, which causes the print execution unit 160 to print the Type 2 designated images, includes the process of printing multiple S325 processes, that is, the process of printing multiple Type 2 designated images IM4-IM6, which are multiple images specified by the user, to the print execution unit 160.

[0098] The controller system 1000c executes the S150 process multiple times. Through the multiple S150 processes, the controller system 1000c generates multiple Type 1 individual vectors Vi1-Vi3, each associated with a multiple Type 1 designated image IM1-IM3 shown in Figure 6(E). Here, the controller system 1000c uses the multiple Type 1 designated images IM1-IM3 and the vector generation model MV. The vector generation model MV is a machine learning model trained to generate embedding vectors of input information. The multiple S150 processes are an example of a first embedding generation process that generates multiple Type 1 individual vectors. In S425 of Figure 4, the controller system 1000c generates a Type 1 vector V1 using the multiple Type 1 individual vectors Vi1-Vi3. The S425 process is an example of a Type 1 vector generation process that generates a Type 1 vector using multiple Type 1 individual vectors.

[0099] The controller system 1000c executes the S350 process multiple times. Through the multiple S350 processes, the controller system 1000c generates multiple Type 2 individual vectors Vi4-Vi6, each associated with one of the multiple Type 2 designated images IM4-IM6 shown in Figure 6(E). Here, the controller system 1000c uses the multiple Type 2 designated images IM4-IM6 and the vector generation model MV. The multiple S350 processes are an example of a second embedding generation process that generates multiple Type 2 individual vectors. In S445 of Figure 4, the controller system 1000c generates a Type 2 vector V2 using the multiple Type 2 individual vectors Vi4-Vi6. The S445 process is an example of a Type 2 vector generation process that generates a Type 2 vector using multiple Type 2 individual vectors.

[0100] With this configuration, the controller system 1000c can generate interest level information D based on the changes between the trends of multiple Type 1 designated images IM1-IM3 printed before the printing of a specific content image IMc, and the trends of multiple Type 2 designated images IM4-IM6 printed after the printing of the specific content image IMc. Therefore, the controller system 1000c can generate interest level information D that appropriately indicates the level of user interest.

[0101] Furthermore, in this embodiment, as shown in Figures 6(B) and 6(C), the vector generation model MV may be either the vector generation model MVt or the vector generation model MVit. That is, the vector generation model MV may include a text embedding model MEt, which is a model trained to generate an embedding vector Vt of the input text TX. When the vector generation model MV includes the text embedding model MEt, the multiple S150 operations, which are an example of the first embedding generation process, include the process of generating multiple individual first-type vectors Vi1-Vi3 using multiple texts represented by multiple first-type designated images IM1-IM3 and the vector generation model MV. Similarly, the multiple S350 operations, which are an example of the second embedding generation process, include the process of generating multiple individual second-type vectors Vi4-Vi6 using multiple texts represented by multiple second-type designated images IM4-IM6 and the vector generation model MV.

[0102] With this configuration, the controller system 1000c can generate interest level information D, taking into account the text contained in the image. When the user's interest is influenced by the text contained in the image, the controller system 1000c can generate interest level information D that appropriately indicates the level of the user's interest.

[0103] Furthermore, in this embodiment, as shown in Figure 7, the controller system 1000c manages print information PF for each user identifier IDu. That is, for each of the multiple user identifiers that identify multiple users, for example, user identifiers IDu1 and IDu2 shown in Figure 7, the controller system 1000c executes the first print process (S125) shown in Figure 2, the content print process (S265) shown in Figure 3, and the second print process (S325) shown in Figure 3. In addition, the interest level process S410, which includes the first decision process (S460), the second decision process (S465), and the storage process (S475) shown in Figure 4, is executed for each combination of printer identifier IDd and user identifier IDu. That is, the controller system 1000c executes the first decision process (S460), the second decision process (S465), and the storage process (S475) for each of the multiple user identifiers. Therefore, by referring to the interest level information D for each user identifier IDu, it is possible to evaluate the user's level of interest in a specific content image for each user identifier IDu.

[0104] In this embodiment, the server processor 310 of the service server 300 performs the following processing according to the program PG3. Specifically, in S460 of Figure 4, the server processor 310 determines the first type similarity SM1 using the first type vector V1 and the specific vector Vc. The first type vector V1 represents the features of the first type designated images IM1-IM3. The specific vector Vc represents the features of the specific content image IMc. The first type similarity SM1 represents the similarity between the first type designated images IM1-IM3 and the specific content image IMc. The processing in S460 is an example of a first determination process that determines the first type similarity between the first type designated images and the specific content image using the first type vector representing the features of the first type designated images and the specific vector representing the features of the specific content image. In S465, the server processor 310 determines the second type similarity SM2 using the second type vector V2 and the specific vector Vc. The second type vector V2 represents the features of the second type designated images IM4-IM6. The second type similarity SM2 represents the similarity between the second type designated images IM4-IM6 and the specific content image IMc. The process in S465 is an example of a second determination process that determines the second type similarity between the second type designated images and the specific content image using the second type vector representing the features of the second type designated images and the specific vector. In S475, the server processor 310 stores the level of interest information D, determined using the first type similarity SM1 and the second type similarity SM2, in the storage device 315, in this case the record database 420 shown in Figure 7. The level of interest information D indicates the level of user interest in the specific content image IMc. The process in S475 is an example of a storage process that stores the level of interest information in the storage device. With this configuration, the level of user interest in the specific content image IMc can be evaluated by referring to the level of interest information D. The service server 300 is an example of an information processing device that performs such processing.

[0105] Here, the first kind vector V1, the specific vector Vc, and the second kind vector V2 may be generated by a device other than the information processing device. Examples of other devices include a server other than the service server 300, or a printer 100. Alternatively, the information processing device may generate vectors V1, Vc, and V2. Here, the first kind individual vectors Vi1-Vi3 and the second kind individual vectors Vi4-Vi6 may be generated by a device other than the information processing device. Instead, the information processing device may generate the individual vectors Vi1-Vi3 and Vi4-Vi6.

[0106] B. Second example: Figure 8 shows another embodiment of content image distribution printing. The figure shows a sequence diagram of the printing system processing. In this embodiment, as with the embodiments in Figures 2-4, the first type user processing S110, content processing S210, second type user processing S310, and interest level processing S410 are performed in order. Unlike the embodiment described in Figure 7, in this embodiment, multiple content images are printed by distribution printing. Then, the interest level information D for each of the printed content images is determined. For example, as shown in Figure 8, in S265 included in content processing S210, multiple content images IMca-IMcd may be printed on a single sheet SH. Also, content processing S210 may be performed multiple times within a short period. For example, the distribution date and time DT shown in Figure 1 may be set to four times on the same day of the week. Examples of the four times include 10:00, 10:05, 10:10, and 10:15. In this embodiment, multiple content images to be printed within a specific period 210t are processed together. The period 210t may be various times, such as 30 minutes.

[0107] Figure 9 shows an example of information recorded in the record database 420. Unlike the example in Figure 7, the record database 420 in Figure 9 records four print information PFs for four content images, each represented by four print identifiers IDpca-IDpcd. These content images are printed on a single sheet SH, as shown in Figure 8, and the time T is set to the same time Tc. In S250 of Figure 3, which is included in the content processing S210 of Figure 8, a specific vector is generated for each content image. The record database 420 records the specific vectors Vca-Vcd and content identifiers IDca-IDcd for each content image. Note that the print information PFs associated with print identifiers IDp1-IDp3 and the print information PFs associated with print identifiers IDp4-IDp6 are the same as those in Figure 7.

[0108] Figure 10 is a block diagram representing the calculation of similarity. The figure shows the first type designated images IM1-IM3, the content images IMca-IMcd associated with the specific vectors Vca-Vcd, the second type designated images IM4-IM6, the vector generation model MV, the individual vectors Vi1-Vi6, the specific vectors Vca-Vcd, the first type vector V1, and the second type vector V2. The difference from the calculation example in Figure 6(E) is that the similarity of each of the four specific vectors Vca-Vcd is calculated.

[0109] In steps S460-S475 of Figure 4, which are included in S410 of Figure 8, the server processor 310 processes each of the specific vectors Vca and Vcd. The first kind vector V1 and the second kind vector V2 are used in common for the specific vectors Vca and Vcd.

[0110] The first-type similarity SM1a shown in Figure 10 is the similarity between the first-type vector V1 and the embedded vector Vca, and the second-type similarity SM2a is the similarity between the second-type vector V2 and the embedded vector Vca. Similarly, the first-type similarities SM1b-SM1d are the similarities between the first-type vector V1 and the specific vectors Vcb-Vcd, respectively, and the second-type similarities SM2b-SM2d are the similarities between the second-type vector V2 and the specific vectors Vcb-Vcd, respectively. In S470 in Figure 4, the server processor 310 determines the interest level information Da-Dd for each of the four content images IMca-IMcd. In S475, each interest level information Da-Dd is recorded in the record database 420 shown in Figure 9. By referring to the interest level information Da-Dd, it is possible to evaluate the user's level of interest in each of the content images IMca-IMcd.

[0111] As described above, in this embodiment, the controller system 1000c may print multiple content images IMca-IMcd on a single sheet SH in S265, which is included in the content processing S210 in Figure 8. Furthermore, the controller system 1000c may execute the content processing S210 multiple times within a specific period 210t. Through such processing, the controller system 1000c causes the print execution unit 160 to print multiple specific content images, in this case, content images IMca-IMcd, that are different from the first designated images IM1-IM3, within a specific period 210t. Thus, the content printing process that causes the print execution unit 160 to print specific content images includes one or more S265 operations that print multiple content images on a single sheet. In other words, the content printing process includes the process of causing the print execution unit 160 to print multiple specific content images, that are different from the first designated images, within a specific period 210t.

[0112] The controller system 1000c executes steps S460-S475 in Figure 4, which are included in step S410 in Figure 8, for each of the multiple specific content images IMca-IMcd. Specifically, the controller system 1000c executes a first determination process (S460) that uses a common first-type vector V1, a second determination process (S465) that uses a common second-type vector V2, and a storage process (S475) for each of the multiple specific content images IMca-IMcd. In this way, since multiple first-type similarity values ​​SM1a-SM1d and multiple second-type similarity values ​​SM2a-SM2d are processed using a common first-type vector V1 and second-type vector V2, the processing burden is reduced compared to the case where a first-type vector V1 and second-type vector V2 are calculated for each specific content image.

[0113] C. Third embodiment: Figure 11 is a diagram illustrating another embodiment of the record database. The difference from the record database 420 in Figures 7 and 9 is that the user identifier IDu is omitted. In this embodiment, the record database 420z records print information PF for each printer identifier IDd. In the example in Figure 11, the record database 420z includes sub-databases 420za-420zd with different printer identifier IDds. Such a record database 420z is applicable to all of the above embodiments. For example, in S155 (Figure 2), S255 (Figure 3), S355 (Figure 3), and S475 (Figure 4), the association with the user identifier is omitted.

[0114] Thus, in this embodiment, the controller system 1000c manages print information PF for each printer identifier IDd. That is, for each of the multiple printer identifiers, for example, printer identifiers IDd1-IDd4 shown in Figure 11, the controller system 1000c performs a first print process (Figure 2: S125), a content print process (Figure 3: S265), and a second print process (Figure 3: S325). Furthermore, the interest level process S410, which includes the first decision process (S460), the second decision process (S465), and the storage process (S475) shown in Figure 4, is performed for each printer identifier IDd. That is, the controller system 1000c performs the first decision process (S460), the second decision process (S465), and the storage process (S475) for each of the multiple printer identifiers IDd. Therefore, by referring to the interest level information D for each printer identifier IDd, it is possible to evaluate the user's level of interest in a specific content image for each printer.

[0115] D. Fourth embodiment: Figure 12 is a diagram illustrating another embodiment of the recording database. Recording database 420x, like recording database 420 in Figure 7, has sub-databases for each combination of printer identifier IDd and user identifier IDu. The diagram shows sub-database 420ax, which is associated with the combination of printer identifier IDd1 and user identifier IDu1, and sub-database 420bx, which is associated with the same printer identifier IDd1 and another user identifier IDu2, among the multiple sub-databases of recording database 420x. The information recorded in sub-database 420ax is the same as the information recorded in sub-database 420a in Figure 7. Print identifiers IDp11-IDp13 in sub-database 420bx indicate print information PF for a Type 1 designated image that is different from the Type 1 designated image in sub-database 420ax, and print identifiers IDp14-IDp16 indicate print information PF for a Type 2 designated image that is different from the Type 2 designated image in sub-database 420ax.

[0116] The difference between the record database 420x in this embodiment and the record database 420 in Figures 7 and 9 and the record database 420z in Figure 11 is that when printing content images, information related to the printing of content images is recorded in a sub-database of all user identifiers IDu associated with the printer identifier IDd of the printer that performed the printing. In the content processing S210 of this embodiment, as described below, S220 and S255 are performed considering all user identifiers IDu associated with the printer identifier IDd of the target printer for distribution printing, unlike in the first embodiment. The processing of the other steps of content processing S210 is the same as the processing of the corresponding steps in the first embodiment.

[0117] In S220 of Figure 3, the server processor 310 obtains the printer identifier IDd of the printer 100 to which the content image will be distributed for printing from the service database 410 shown in Figure 1. Various images may be obtained as the content image. For example, the server processor 310 may obtain a content image as in the first embodiment.

[0118] In S255 of Figure 3, the server processor 310 selects all sub-databases of user identifiers IDu associated with the acquired printer identifier IDd from the recording database 420x shown in Figure 12, and records the print information PF in all of the selected sub-databases by associating a specific vector, content identifier IDc, and time T. The print information PF shown by the print identifiers IDpc and IDpcx in the sub-databases 420ax and 420bx in Figure 12 is an example of the print information PF recorded in S255.

[0119] Thus, in this embodiment, the controller system 1000c records print information PF related to the printing of content images for all user identifiers IDu associated with the printer identifier IDd. The interest level processing S410 is performed for all user identifiers IDu associated with the printer identifier IDd. In the interest level processing S410, the controller system 1000c performs the same processing as in the first embodiment. As a result, interest information D of content images can be recorded for each of the multiple users who share a particular printer. As shown in Figure 12, the print identifiers IDpc and IDpcx of the sub-databases 420ax and 420bx of user identifiers IDu1 and IDu2 associated with the printer identifier IDd1 record the results of the calculation of the change in similarity between the specified image and the content image in each sub-database. In the example in Figure 12, different interest levels D1 and D2 are recorded in the print information PF of the print identifiers IDpc and IDpcx of the sub-databases 420ax and 420bx. Therefore, by comparing the interest level information D of all user identifiers IDu associated with a particular printer identifier IDd, it is possible to evaluate each user's level of interest in a particular content image.

[0120] Furthermore, the printer identifier IDd may be associated with the user identifier IDu of the user who has contracted the print distribution service, as well as the user identifier IDu of other users who use the same printer. For example, in the contract process for the print distribution service, the contract holder may apply for other users, such as family members, to use the same printer in addition to themselves. In response to this application, the service database 410 in Figure 1 may record the correspondence between the printer identifier IDd and multiple user identifier IDu. The delivery date and time DT of users other than the contract holder may be set to the contract holder's delivery date and time DT, or it may not be set.

[0121] E. Variations: (1) The method for extracting text from the image shown in S530 of Figure 5 is not limited to the method using the text extraction model Mtxt, but may be any method. For example, the text may be extracted by template matching using template images that represent characters.

[0122] (2) The image embedding model MEi is not limited to a convolutional neural network, but may be any model capable of generating vectors representing the features of an input image. For example, the image embedding model MEi may be a model formed by multiple fully connected layers. In any case, the image embedding model MEi may be trained to classify multiple types of images. A model trained in this way can generate vectors representing the features of an image.

[0123] (3) The text embedding model MEt may be any model capable of generating vectors representing the features of the input text. For example, a text embedding model provided by OpenAI may be used. Alternatively, a model formed by multiple fully connected layers may generate vectors from a sequence of tokens representing a sequence of words.

[0124] (4) The method for obtaining the embedding vector is not limited to using a trained machine learning model, but may be any of the various other methods. For example, an array of multiple color values ​​of multiple pixels evenly extracted from the entire image may be used as the embedding vector associated with the image. Alternatively, a technique called Bag of Words (BoW) may be used, which counts the frequency of occurrence of each of several specific words in the text and uses the frequencies to generate an embedding vector associated with the text. Here, a technique called TF-IDF (Term Frequency-Inverse Document Frequency) may be used to vectorize the text in a way that reflects the importance of the words in the text.

[0125] (5) As explained in Figure 5, any of the three types of embedding vectors may be used: image embedding vectors, text embedding vectors, and image-text embedding vectors. There may be various methods for selecting the type of embedding vector. For example, a predetermined type of embedding vector may be used.

[0126] Furthermore, the server processor 310 may select the type of embedding vector according to the number of characters in the text extracted from the specified image, for example, the average number of characters in multiple specified images. For example, if the number of characters is greater than or equal to the first threshold, a text embedding vector may be selected; if the number of characters is less than the first threshold but greater than or equal to the second threshold, an image-text embedding vector may be selected; and if the number of characters is less than the second threshold, an image embedding vector may be selected. Here, the thresholds have the relationship first threshold > second threshold > zero. The server processor 310 may perform the calculation of the number of characters and the selection of the type of embedding vector in the interest processing S410 shown in Figure 4. In order to allow any type to be selected, it is preferable that the server processor 310 generates three types of embedding vectors in S150 in Figure 2, S250 in Figure 3, and S350 in Figure 3. It is preferable that the three types of embedding vectors are recorded in the recording database 420 shown in Figures 7 and 9, the recording database 420z shown in Figure 11, and the recording database 420x shown in Figure 12.

[0127] (6) For example, the method for generating the first kind vector V1 shown in S425 of Figure 4 is not limited to a method for generating a vector that represents the centroid of multiple individual first kind vectors, but may be various methods for generating a vector that represents multiple individual first kind vectors. For example, a vector randomly selected from multiple individual first kind vectors may be used as the first kind vector V1. This first kind vector V1 may also change if any one of the multiple individual first kind vectors changes. Therefore, it can be said that such a first kind vector V1 represents the respective features of multiple designated first kind images. For example, the method for generating the second kind vector V2 in S445 of Figure 4 may be the same as the method for generating the first kind vector V1.

[0128] (7) For example, the method for calculating the similarity SM1 of S460 in Figure 4 and the similarity SM2 of S465 in Figure 4 is not limited to the method for calculating cosine similarity, but may be any of the various methods for calculating a value that represents the similarity between two vectors. For example, the similarity may be calculated as a distance such as the Euclidean distance, Manhattan distance, or Max distance. When similarity represents distance, the shorter the distance, the higher the similarity.

[0129] (8) When information Da representing a combination of similarity scores SM1 and SM2 is used as interest level information D, in S475 of Figure 4, the server processor 310 may record the set of similarity scores SM1 and SM2 calculated in S460 and S465 as interest level information D in the record database 420 shown in Figures 7 and 9, the record database 420z shown in Figure 11, or the record database 420x shown in Figure 12. In this case, interest level information D is determined by the process of recording the set of similarity scores SM1 and SM2 as interest level information D. Therefore, S470 may be omitted.

[0130] (9) The method for acquiring advertising images is not limited to auctions and may be various other methods. For example, in S220 of Figure 3, the server processor 310 may acquire the advertising image to be printed from a plurality of pre-prepared advertising images.

[0131] (10) In a narrow sense, advertising may mean informing people for the purpose of promoting goods or services, but in this specification, advertising means in general any form of informing people (in this embodiment, printer users) by paying advertising fees, not limited to promotional purposes. Advertising image means an image that shows the information to be advertised. Accordingly, in this specification, advertising image is not limited to an image that shows information intended to promote goods or services provided by the advertiser, but includes images that show various kinds of information provided by the advertiser (e.g., weather information, traffic information, the advertiser's own works). Furthermore, advertising image may be any image that a third party other than the printer user, such as the advertiser, wants to print on the user's printer.

[0132] The content is not limited to advertising content; it can be any content. For example, the server processor 310 may obtain content images from an image collection provided by a person related to the printer user, such as a family member, relative, or friend. The image collection may include various images, such as photographs and paintings. The person related to the printer user can evaluate the user's level of interest in the content images by referring to the interest level information D in the record database 420 shown in Figures 7 and 9, the record database 420z shown in Figure 11, or the record database 420x shown in Figure 12.

[0133] (11) The method for identifying the user who issued the print command is not limited to the method of referring to a user logged into the printer as shown in S135 of Figure 2, but may be any of the following methods. For example, the printer may have a camera that takes a picture of the face of the user operating the printer. A facial image may be associated with each of the multiple user identifiers IDu. The printer processor 110 may take a picture of the face while printing is being performed at the user's command. The printer processor 110 may use a user identifier IDu associated with the face represented by the captured image.

[0134] (12) The storage device that stores the interest level information D, for example, the record database 420 shown in Figures 7 and 9, the record database 420z shown in Figure 11, or the record database 420x shown in Figure 12, may be an external storage device connected to the printing system 1000 instead of the internal storage device 315 of the service server 300. For example, the record databases 420, 420z, and 420x may be stored in a storage server (not shown) connected to the network IT.

[0135] (13) The configuration of the service server may be any of the various configurations shown for the service server 300 in Figure 1. For example, multiple devices (e.g., computers) that can communicate with each other via a network may each share some of the data processing functions of the service server, and together they may provide the functions of the service server. A system equipped with these devices corresponds to a service server.

[0136] (14) In each of the above embodiments, the printer 100 and the service server 300 each perform a portion of the printing system processing shown in Figures 2-4. The printer 100 may perform some or all of the processing performed by the service server 300. For example, in the interest level processing S410 in Figure 4, the printer 100 may perform a first determination process (S460) to determine the first type of similarity, a second determination process (S465) to determine the second type of similarity, and a storage process (S475) to store the interest level information in a storage device. In this case, the printer 100 is an example of an information processing device that performs these processes. Here, the printer processor 110 may perform these processes according to the program PG1.

[0137] Note that the first kind vector V1, the specific vector Vc, and the second kind vector V2 may be generated by a device other than the information processing device, for example, a service server 300. Alternatively, the information processing device may generate vectors V1, Vc, and V2. Here, the first kind individual vectors Vi1-Vi3 and the second kind individual vectors Vi4-Vi6 may be generated by a device other than the information processing device. Alternatively, the information processing device may generate individual vectors Vi1-Vi3 and Vi4-Vi6.

[0138] Furthermore, the server may be omitted from the printing system 1000. In other words, the printer may perform all the processing. For example, the printer processor 110 of printer 100 may perform the printing system processing shown in Figures 2-4 according to program PG1.

[0139] (15) The information recorded in the record database 420 shown in Figures 7 and 9, the record database 420z shown in Figure 11, or the record database 420x shown in Figure 12 may be various types of information. For example, the time T may be omitted. The print type PT may be omitted. In addition, record databases 420, 420z, and 420x may be provided for each printer. In this case, the printer identifier IDd in record databases 420, 420z, and 420x may be omitted.

[0140] (16) The printer configuration may be different from that of printer 100 in Figure 1, for example, a controller system including one or more controllers such as a CPU or SoC and a print execution unit. For example, the reading execution unit 170 may be omitted. Also, the sheet used for printing may be various sheet-like media such as film instead of paper.

[0141] In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the processing performed by the printer may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0142] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in a form stored on a non-temporary computer-readable recording medium. The program may be used while stored on the same or a different computer-readable recording medium as at the time of provision. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0143] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of Symbols]

[0144] 100…Printer, 110…Printer processor, 115,315…Storage device, 120,320…Volatile storage device, 130,330…Non-volatile storage device, 140…Display unit, 150…Operation unit, 160…Print execution unit, 170…Read execution unit, 180,380…Communication interface, 210t…Specific period, 300…Service server, 310…Server processor, 410…Service database, 420,420z…Recording database, 420a-420d,420za-420zd…Sub-database, 1000…Printing system, 1000c…Controller system, IT…Network

Claims

1. A printing system, Print execution unit, A controller system including one or more controllers, Equipped with, The controller system is A first printing process is executed to cause the print execution unit to print a first type designated image, which is an image specified by the user. After the first printing process, a content printing process is executed to cause the printing execution unit to print a specific content image different from the first designated image. After the content printing process, a second printing process is executed to cause the printing execution unit to print a second designated image, which is an image different from the specific content image and is an image specified by the user. A first determination process is performed to determine a first-type similarity, which represents the similarity between the first-type designated image and the specific content image, using a first-type vector representing the features of the first-type designated image and a specific vector representing the features of the specific content image. A second determination process is performed to determine a second-type similarity, which represents the similarity between the second-type designated image and the specific content image, using a second-type vector representing the features of the second-type designated image and the specific vector. The system performs a storage process to store interest information, which is determined using the first type of similarity and the second type of similarity, and which indicates the level of interest of the user in the specific content image, in a storage device. Printing system.

2. A printing system according to claim 1, The controller system is An interest level determination process is performed to determine the interest level information that represents the difference between the second type of similarity and the first type of similarity, or the sign of the difference. Printing system.

3. A printing system according to claim 1 or 2, The first printing process includes causing the printing execution unit to print a plurality of first-type designated images, which are a plurality of images specified by the user. The second printing process includes causing the printing execution unit to print a plurality of second-type designated images, which are a plurality of images specified by the user. The controller system is Using the aforementioned plurality of designated first-type images and a machine learning model trained to generate embedding vectors for input information, a first embedding generation process is performed to generate a plurality of individual first-type vectors, which are a plurality of embedding vectors associated with each of the plurality of designated first-type images. Using the plurality of individual first-kind vectors, a first-kind vector generation process is executed to generate the first-kind vector, Using the aforementioned plurality of Type 2 designated images and the machine learning model, a second embedding generation process is executed to generate a plurality of Type 2 individual vectors, which are a plurality of embedding vectors associated with each of the plurality of Type 2 designated images. Using the aforementioned plurality of separate second-kind vectors, a second-kind vector generation process is performed to generate the second-kind vector. Printing system.

4. A printing system according to claim 3, The aforementioned machine learning model includes a model trained to generate an embedding vector for the input text, The first embedding generation process includes a process for generating the plurality of individual first vectors using the plurality of texts represented by the plurality of first designated images and the machine learning model, The second embedding generation process includes a process of generating the plurality of individual second-type vectors using the plurality of texts represented by the plurality of second-type designated images and the machine learning model. Printing system.

5. A printing system according to claim 1 or 2, The controller system is For each of the multiple user identifiers that identify multiple users, the first printing process, the content printing process, the second printing process, the first determination process, the second determination process, and the storage process are executed. Printing system.

6. A printing system according to claim 1 or 2, The controller system is For each of the multiple printer identifiers that identify multiple printers, each having a print execution unit, the first print process, the content print process, the second print process, the first determination process, the second determination process, and the storage process are executed. Printing system.

7. A printing system according to claim 1 or 2, The content printing process includes a process that causes the printing execution unit to print a plurality of specific content images different from the first designated image within a specific period of time. The controller system, for each of the plurality of specific content images, The first decision process which uses the first kind vector in common, The second decision process which uses the aforementioned second kind vector in common, The aforementioned storage process, A printing system that executes this.

8. An information processing device, A first decision processing unit performs a first decision process to determine a first type similarity, which represents the similarity between a first type designated image, which is an image specified by a user, and a specific content image that is different from the first type designated image, using a first type vector representing the features of the first type designated image and a specific vector representing the features of the specific content image. A second decision processing unit performs a second decision process to determine a second type similarity, which represents the similarity between a second type designated image, which is an image different from the aforementioned specific content image and an image specified by the user, and the aforementioned specific content image, using a second type vector representing the features of the second type designated image and the aforementioned specific vector. A storage processing unit that performs a storage process to store interest level information, which is determined using the first type of similarity and the second type of similarity, and which indicates the level of interest of the user to the specific content image, in a storage device. An information processing device equipped with the following features.

9. It is a program, A first determination processing function that performs a first determination process to determine a first-type similarity, which represents the similarity between a first-type designated image, which is an image specified by the user, and a specific content image different from the first-type designated image, using a first-type vector representing the features of the first-type designated image and a specific vector representing the features of the specific content image. A second determination processing function performs a second determination process to determine a second type similarity, which represents the similarity between a second type designated image (an image different from the aforementioned specific content image and specified by the user) and the aforementioned specific content image, using a second type vector representing the features of the second type designated image and the aforementioned specific vector. A storage processing function that performs a storage process to store interest level information, which is determined using the first type of similarity and the second type of similarity, and which indicates the level of interest of the user to the specific content image, in a storage device. A program that enables a computer to realize something.

10. Information processing method, The first printing process is executed, causing the print execution unit to print the first type of designated image, which is an image specified by the user. After the first printing process, a content printing process is executed to cause the printing execution unit to print a specific content image different from the first designated image. After the content printing process, a second printing process is executed to cause the printing execution unit to print a second designated image, which is an image different from the specific content image and is an image specified by the user. A first determination process is performed to determine a first-type similarity, which represents the similarity between the first-type designated image and the specific content image, using a first-type vector representing the features of the first-type designated image and a specific vector representing the features of the specific content image. A second determination process is performed to determine a second-type similarity, which represents the similarity between the second-type designated image and the specific content image, using a second-type vector representing the features of the second-type designated image and the specific vector. The system performs a storage process to store interest information, which is determined using the first type of similarity and the second type of similarity, and which indicates the level of interest of the user in the specific content image, in a storage device. Information processing methods.

Images