Image processing system

Abstract

To reduce the time and effort for a user to generate stamp data conforming to guidelines.SOLUTION: An image processing system has: generation means that generates stamp data available in a messenger application by performing image processing on photographed image data; and determination means that determines whether the stamp data generated by the generation means conforms to guidelines.SELECTED DRAWING: Figure 15

Description

[[Technical Field]]

[0001] The present disclosure relates to an image processing system, a processing method for an image processing system, and a program. [[Background Art]]

[0002] In recent years, with the popularization of information processing terminals such as smartphones that are capable of capturing still images and moving images and have a communication function, various types of communication using images in addition to voice and text have become common.

[0003] As communication application software that operates on information processing terminals, there are messenger apps. In messenger apps, in addition to text messages, image data called stamps or stickers that express words often used in daily conversation, emotional expressions, and reactions with illustrations are used to expand the scope of communication and allow users to enjoy interacting with each other. Stamp data is provided as a default data set included in the aforementioned messenger app, in an environment where several pre-prepared patterns of stamps can be used.

[0004] In addition to this, when a user accesses a predetermined online store linked to the messenger app and purchases stamp data, it is possible to add and use various stamps that suit the user's own preferences. Furthermore, by passing a predetermined review process defined by the company operating the messenger app, individual users can sell original stamps created by themselves on the online store.

[0005] On the other hand, in order for an individual user to create and sell original stamps, it may be necessary to separately purchase and prepare dedicated equipment such as drawing software and a pen tablet. In addition, the number of sets of patterns required for listing on an online store is as large as several tens of sheets, and it is necessary to pass a review to confirm compliance with guidelines, so multiple processes must be gone through.

[0006] Therefore, it takes a long time to prepare all the designs, and depending on the content of the designs, they may be rejected and sent back for review, requiring revisions and resubmission, resulting in a high barrier to entry for beginner users.

[0007] Patent Document 1 discloses a method for forming a face image that conforms to the size of a face photograph specified in various application forms and cards in a single photograph by photographing the subject's face and the background surrounding the face, and then cropping the background around the face in the photographed data to a predetermined size. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2001-8018 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] By applying the method described in Patent Document 1, it is possible to eliminate the need for users to create illustrations from scratch and to obtain image data for predetermined use that complies with the guidelines.

[0010] However, Patent Document 1 only provides a means for automatically correcting the field of view to a predetermined angle. Patent Document 1 does not include the conversion process for stamp illustration data mentioned above, nor does it conform the data content to a format compliant with the guidelines, thereby streamlining the review flow up to sales. For this reason, the method of Patent Document 1 still leaves the problem that it is still time-consuming for an individual user to create and sell original stamps.

[0011] The purpose of this disclosure is to reduce the effort required of users to generate stamp data that conforms to the guidelines. [Means for solving the problem]

[0012] The image processing system includes a generation means for generating stamp data usable in a messenger app by processing captured image data, and a determination means for determining whether the stamp data generated by the generation means conforms to the guidelines. The system includes, when the number of stamp data determined by the determination means to conform to the guidelines reaches a predetermined number required for a sales review request to the messenger app operating website, a request means for submitting a sales review request to the messenger app operating website for the predetermined number of stamp data determined to conform to the guidelines, and a display control means that controls the system to display the number of remaining stamp data required until the number of stamp data determined by the determination means to conform to the guidelines reaches the predetermined number. to have The generation means generates a plurality of stamp data, and the determination means determines whether each of the plurality of stamp data generated by the generation means conforms to the guidelines. . [Effects of the Invention]

[0013] According to this disclosure, the effort required of users to generate stamp data that conforms to the guidelines can be reduced. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram showing an example of the configuration of an image processing system. [Figure 2] This figure shows an example of the configuration of an imaging device. [Figure 3] This figure shows an example of a server device configuration. [Figure 4] This is a flowchart showing the processing method of the imaging device. [Figure 5] This is a flowchart showing the processing method of the server device. [Figure 6] This figure shows an example of the pre-approval menu screen for an imaging device. [Figure 7] This is a diagram showing an example of the stamp illustration style selection screen. [Figure 8] This figure shows an example of the remaining number of frames displayed on an imaging device. [Figure 9] This figure shows an example of voice-guided imaging using an imaging device. [Figure 10] This figure shows an example of guided shooting using the live view display of the imaging device. [Figure 11] This figure shows an example of the stamp preview operation of an imaging device. [Figure 12] This figure shows an example of a notification menu for failing the inspection of an imaging device. [Figure 13] This is a diagram illustrating an example operation of re-learning a network model using an examination result. [Figure 14] This is a flowchart illustrating a processing method of an image capturing apparatus. [Figure 15] This is a flowchart illustrating a processing method of a server apparatus. MODE FOR CARRYING OUT THE INVENTION

[0015] (First Embodiment) Hereinafter, the first embodiment will be described in detail with reference to the drawings. Note that the following embodiments do not limit the scope of the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential, and the plurality of features may be arbitrarily combined. Furthermore, in the drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate description is omitted.

[0016] <System Configuration> FIG. 1 is a diagram illustrating a configuration example of an image processing system 100 according to the first embodiment. The image processing system 100 includes an image capturing apparatus 200, a server apparatus 300, a database 400, a server apparatus 500, a mobile terminal 600, and a network 700. The network 700 is, for example, the Internet. These apparatuses are mutually connected via the network 700, and transmit and receive control signals and data between the apparatuses. Depending on requirements for confidentiality, secrecy or allowable transmission delay of handled data, connection may be made via a dedicated line or physical cables.

[0017] The imaging device 200 has, in addition to its shooting function, a display function that displays and outputs various settings during shooting and image data after shooting based on user operation, and a voice guidance function that provides voice guidance to the photographer and the main subject using a built-in speaker or the like. Furthermore, the imaging device 200 has the function of connecting to the network 700 using a public wireless communication network or a wireless communication method such as Wi-Fi, and transmitting and receiving data with the server devices 300 and 500. Typical forms of the imaging device 200 include digital cameras and video camcorders, but it is not limited to devices with similar functions.

[0018] The server device 300 is a server device for checking compliance with stamp guidelines. After receiving stamp data created by applying image processing to image data captured by the imaging device 200, the server device 300 analyzes and checks the stamp data based on predetermined inspection rules and determines whether or not it is possible to request stamp sales review from the server device 500.

[0019] Here, the inspection rules described above apply the requirements that must be met when selling stamps, as defined by the operator managing and operating server device 500, as the basic rules, and also include checks based on the results of the review of stamp data previously submitted for sale.

[0020] The server device 300 stores the stamp data review results in a large-capacity database 400 connected to the server device 300. Then, for stamp data that is deemed unsaleable based on the review results, the server device 300 sends the review results to the imaging device 200 to prompt re-shooting and creation of new stamp data.

[0021] On the other hand, if the server device 300 determines that the stamps are for sale and has completed the generation of a predetermined number of stamps, it sends the stamp data along with the account information of the stamp-creating user to the server device 500 and processes the sales application on behalf of the user.

[0022] Server device 500 is a server device for distributing social networking services (hereinafter referred to as SNS). Server device 500 is a server device that implements a community-type membership service that promotes and supports connections between people on the web, or a website that provides such a service. Furthermore, server device 500 is operated as a server that performs sales review of SNS applications designated by the business operator, or it is assumed that it also includes website functionality.

[0023] For example, SNS include LINE (registered trademark), Facebook Messenger (registered trademark), Snapchat (registered trademark), WhatsApp (registered trademark), KakaoTalk (registered trademark), or WeChat (registered trademark). SNS are not limited to services that enable user-to-user communication, sales, and review functions using the aforementioned stamps.

[0024] Furthermore, in order to realize the above services, the server device 500 can send and receive data to and from the mobile terminal 600 connected to the network 700, including stamp data transmitted from the server device 300.

[0025] The mobile terminal 600 is a device owned by the same user as the imaging device 200, and like the imaging device 200, it has the function of connecting to the network 700 using a wireless communication method and is equipped with an application program that uses SNS (hereinafter referred to as SNS app). Through the SNS app, the mobile terminal 600 can use stamp data generated from image data captured by the imaging device 200 to communicate with many users via the server device 500. In addition, the mobile terminal 600 receives the results of the stamp sales review request via the server device 300. The mobile terminal 600 is, for example, a smartphone or a tablet device, but is not limited to devices with similar functions.

[0026] The above describes the configuration of the image processing system 100. Next, we will explain the configuration and processing details of the imaging device 200 and the server device 300.

[0027] <Configuration of imaging device 200> Figure 2 shows an example of the hardware configuration of the imaging device 200 shown in Figure 1. The imaging device 200 includes a CPU 201, a primary storage unit 202, an imaging unit 203, an image processing unit 204, a communication unit 205, a recording and playback unit 206, a display unit 207, an audio processing unit 208, and an operation unit 209. Furthermore, the imaging device 200 includes a CPU bus 210, a work memory 211, and a memory bus 212.

[0028] The CPU 201 is the Central Processing Unit. The CPU 201 controls the system via the CPU bus 210, including starting and stopping the imaging unit 203, image processing unit 204, communication unit 205, recording and playback unit 206, display unit 207, audio processing unit 208, and operation unit 209, as well as providing interrupt notifications. The CPU 201 also controls various operations of the imaging device 200.

[0029] The primary storage unit 202 is a storage area used as a work area for the CPU 201, etc. The primary storage unit 202 is implemented by, for example, DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), or non-volatile flash memory. For example, the CPU 201 realizes various functions provided by the imaging device 200 by loading and executing control programs stored in the primary storage unit 202.

[0030] The imaging unit 203 includes a camera unit such as a lens and CCD, and an optical unit. It converts light captured from the lens into an electrical signal using a sensor, and then outputs the digital RAW image data, converted to Bayer format, to a work memory 211 such as a large-capacity DRAM via the memory bus 212.

[0031] The image processing unit 204 performs debayering (demosaicing) on ​​the RAW image data output by the imaging unit 203, converting it into a signal consisting of luminance and chrominance. It then performs so-called development processing to optimize the image by removing noise and correcting optical distortion contained in each signal. Subsequently, the image processing unit 204 compresses and encodes the developed image data according to standardized specifications such as JPEG or HEVC, and outputs the compressed image data to the work memory 211. The image processing unit 204 also reads the compressed data from the recording / playback unit 206, expands it in the work memory 211, and then performs decompression processing again to restore the image data.

[0032] Furthermore, the image processing unit 204 has a function to convert the captured image data after development processing into illustration data for stamps. The image processing unit 204 converts the so-called natural image after capture into a two-dimensional illustration. The conversion method includes color reduction processing by dithering or binarization, gradation correction, various filter processing such as edge filters and smoothing filters, and methods that combine multiple of these to create manga style, anime style, oil painting style, or pixel art style. In addition, the conversion method may also employ methods such as style conversion using content images and style images utilizing DNN (Deep Neural Network), which has attracted attention in recent years, and is not limited to any method that realizes the same image conversion function.

[0033] The image conversion function described above can be implemented using dedicated hardware such as a DSP (Digital Signal Processor) or wiring logic, enabling high-speed real-time processing. However, the image conversion function described above may also be achieved with equivalent functionality and performance through software processing on CPU201, and is not limited to this method.

[0034] The communication unit 205 is equipped with wireless communication interfaces compliant with various communication standards, such as public wireless communication such as LTE and 5G, and wireless LAN (Local Area Network) such as Wi-Fi. The communication unit 205 connects to external devices via the network 700 and transmits and receives image data and other various types of information. Furthermore, the communication unit 205 is equipped with short-range communication interfaces such as infrared communication and Bluetooth, and transmits and receives voice data and control signals to wireless earphones, remote controllers, etc.

[0035] The recording and playback unit 206 is implemented using non-volatile memory such as a USB memory stick, memory card or hard disk drive, or flash memory, and performs writing and reading of various information and data. For example, the recording and playback unit 206 is mainly used as a secondary storage unit to save or play back image data temporarily stored in the work memory 211.

[0036] The display unit 207 has a display device such as an EVF (Electric View Finder) or an LCD (Liquid Crystal Display). The display unit 207 displays graphics data such as characters and menu icons created by the CPU 201, image data and decoded image data developed by the image processing unit 204, and illustration data for stamps on the display device.

[0037] The audio processing unit 208 uses a device such as a speaker built into the imaging device 200 to convert audio data previously stored in the primary storage unit 202 or the work memory 211 from digital to analog and output it.

[0038] The operation unit 209 has operating means including one or more operating members for the user to input various operation instructions. For example, the operation unit 209 detects instructions to switch shooting modes using buttons or switch mechanisms, or detects contact positions using a touch panel integrated with the LCD of the display unit 207, and notifies the CPU 201 of the detected information.

[0039] The CPU bus 210 is a control bus that connects the CPU 201 to each of the processing blocks mentioned above. It may use a standardized bus protocol similar to the memory bus 212, or, if there is sufficient processing capacity, a slower serial protocol such as I2C may be used; it is not limited to any particular protocol.

[0040] The work memory 211, like the primary storage unit 202, is implemented using DRAM or SRAM. The work memory 211 stores data as a temporary working area. The work memory 211 stores RAW image data output by the imaging unit 203, image data input / output by the image processing unit 204, data received from external devices by the communication unit 205, and menu and character data to be superimposed on the image data and output by the display unit 207. The work memory 211 also stores audio data processed by the audio processing unit 208.

[0041] The memory bus 212 connects the imaging unit 203, image processing unit 204, communication unit 205, recording / playback unit 206, display unit 207, audio processing unit 208, and operation unit 209 to the work memory 211, and is a data bus for high-speed transfer of image data and various parameter metadata. The bus transfer method of the memory bus 212 may be a standard bus specification such as ISA, PCI-Express, or AXI, or a proprietary bus method may be adopted; it is not limited to these.

[0042] The above is an example of the hardware configuration of the imaging device 200. Next, the server device 300 will be described.

[0043] <Configuration of Server Device 300> Figure 3 shows an example of the hardware configuration of the server device 300 shown in Figure 1. The server device 300 includes a CPU 301, a primary storage unit 302, an image processing unit 303, a communication unit 304, a recording and playback unit 305, a display unit 306, an operation unit 307, and a machine learning and inference processing unit 308. Furthermore, the server device 300 includes a CPU bus 309, a work memory 310, and a memory bus 311.

[0044] The CPU 301 controls the various operations of the server device 300 by performing system control such as starting and stopping the image processing unit 303, communication unit 304, recording and playback unit 305, display unit 306, operation unit 307, and machine learning and inference processing unit 308, as well as interrupt notifications, via the CPU bus 309. For example, the CPU 301 realizes the various functions provided by the server device 300 by deploying and executing control programs stored in the primary storage unit 302, work memory 310, or recording and playback unit 305.

[0045] The primary storage unit 302 is a memory area used as a work area for the CPU 301, etc. The primary storage unit 302 is implemented by, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory).

[0046] The image processing unit 303 has the function of decoding and analyzing stamp data received from the imaging device 200 via the communication unit 304. In the decoding process, the image processing unit 303 checks whether the file format (file extension) is in accordance with the stamp guidelines set by the operator of the server device 500, and whether it is decodeable. In the analysis process, the image processing unit 303 obtains basic image format information such as the image size, number of horizontal pixels, number of vertical pixels, number of colors, and pixel bit depth of the stamp data.

[0047] Furthermore, the image processing unit 303 separates and extracts the text portion from the non-text portion within the stamp data, for example, using an OCR method. The image processing unit 303 then calculates the correctness of the text content, the ratio of the text portion to the illustration portion generated from the non-text subject within the stamp data, the presence or absence of the illustration portion, and the degree and area of ​​overlap between the illustration portion and the text portion. Finally, the image processing unit 303 calculates the number of colors present in the data, a histogram of pixel values, or the dynamic range across the entire illustration portion, including the background, and the text portion.

[0048] The functions performed by the image processing unit 303 are implemented using dedicated hardware such as a DSP (Digital Signal Processor) or wiring logic, enabling high-speed real-time processing. However, the functions performed by the image processing unit 303 may also be achieved by software processing of the CPU 301, and are not limited to this.

[0049] The communication unit 304 is an interface for connecting the server device 300 with an external device. The server device 300 uses the communication unit 304 to send and receive various information and data to and from the external device via the network 700.

[0050] Network 700 can be, for example, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or a public network. Network 700 is not limited to any method that can establish the transmission and reception of information to and from the imaging device 200, the server device 500, and the mobile terminal 600. Furthermore, network 700 may be a wireless network or a wired network, and may include multiple different types of networks as described above.

[0051] The recording / playback unit 305 is implemented using non-volatile memory such as a hard disk drive or SSD (Solid State Drive), and performs writing and reading of various information, data, or programs. Alternatively, the recording / playback unit 305 may be implemented using a device that writes and reads information, data, or programs to an external recording medium such as an optical disc like a DVD-RW, CD-ROM, CD-R, or DVD-RAM.

[0052] Furthermore, the recording and playback unit 305 is connected to the database 400 shown in Figure 1, and records the stamp inspection results in the database 400, or reads the stamp inspection results from the database 400.

[0053] The display unit 306 is an output interface that displays display information such as images on a screen and presents various information to the user, and is implemented by a display device such as an LCD. The server device 300 does not necessarily have to include the display unit 306, and in the case of a configuration that does not include the display unit 306, it does not have to have a display control function to control the operation of the display unit 306.

[0054] The operation unit 307 is an input interface that receives instructions from the user and can be implemented by, for example, a keyboard, a pointing device, or a touch panel. The server device 300 does not necessarily have to include the operation unit 307, and in configurations that do not include the operation unit 307, it does not need to have an operation unit control function to control the operation of the operation unit 307.

[0055] The machine learning and inference processing unit 308 has the function of training and performing inference on a model composed of a neural network. Based on the stamp data to be reviewed and the stamp review results, the machine learning and inference processing unit 308 retrains the model, infers the content depicted in the stamp data, and outputs a pre-check result of the stamp's compliance with the sales guidelines.

[0056] The CPU bus 309 is implemented with the same configuration as the CPU bus 210 in Figure 2 and has the same functionality.

[0057] The work memory 310, like the primary storage unit 302, is implemented using DRAM or SRAM. The work memory 310 stores image data input / output by the image processing unit 303, data received from external devices by the communication unit 304, and data output by the display unit 306 as a temporary working area.

[0058] Memory bus 311 is implemented with the same configuration as memory bus 212 in Figure 2 and has the same functionality.

[0059] <Operation flow of imaging device 200> Figure 4 is a flowchart showing the processing method of the imaging device 200. The flowchart in Figure 4 is mainly executed by the CPU 201 of the imaging device 200 executing a program.

[0060] First, in step S401, the CPU 201 is triggered by a mode setting from the operation unit 209, which has an interface such as buttons, a directional pad, or a touch panel, and determines whether the operating mode of the imaging device 200 is the shooting mode. If the operating mode is not the shooting mode (NO in step S401), the CPU 201 terminates the processing of the flowchart in Figure 4. If the operating mode is the shooting mode (YES in step S401), the process proceeds to step S402.

[0061] In step S402, the CPU 201 determines whether or not the stamp creation mode has been selected based on user operation of the operation unit 209. If the stamp creation mode has not been selected (NO in step S402), the CPU 201 returns to step S401. If the stamp creation mode has been selected (YES in step S402), the CPU 201 proceeds to step S403.

[0062] In step S403, the CPU 201 activates the stamp creation function of the imaging device 200 and displays the pre-approval menu screens shown in Figures 6(a) to 6(c) on the display unit 207. Specifically, the CPU 201 displays on the display unit 207 the service details and precautions regarding the procedure for automatically generating stamp data from a series of captured images and acting as an agent for sales review requests.

[0063] <Example of pre-approval menu screen> Figures 6(a) to 6(c) show examples of the pre-approval menu screen for the service function displayed in step S403. Figure 6(a) shows the first page of the pre-approval menu screen, Figure 6(b) shows the second page of the pre-approval menu screen, and Figure 6(c) shows the third page of the pre-approval menu screen.

[0064] The pre-consent menu screens in Figures 6(a) to 6(c) show the content that users must agree to in advance before using the service. The final page of Figure 6(c) is a screen where users can confirm whether or not they agree to the content.

[0065] The screen in Figure 6(a) states that the pre-check on the server device 300 is merely a function to make it easier to pass the main review. The screen in Figure 6(b) states that the stamp data submitted for sale and the review results will be used to improve the identification accuracy of machine learning and inference models. Furthermore, the screen in Figure 6(b) states that a portion of the sales revenue from stamps sold by the user after passing the sales review will be returned to the service provider that provides the series of services. Note that the pre-approval menu screen may include other supplementary clauses and restrictions, and is not limited to its wording or expression.

[0066] Returning to Figure 4, in step S404, the CPU 201 determines whether the user has agreed to the service content on the screen shown in Figure 6(c) based on user operation on the operation unit 209. Specifically, the CPU 201 determines whether the user has agreed to the service content based on input from the operation unit 209, such as buttons, directional keys, or touch panel, corresponding to the consent icon on the screen shown in Figure 6(c). If the user has not agreed to the service content (NO in step S404), the CPU 201 returns to step S401. If the user has agreed to the service content (YES in step S404), the CPU 201 proceeds to step S405.

[0067] In step S405, the CPU 201 displays the illustration style selection screen shown in Figure 7 on the display unit 207, which allows the image processing unit 204 to select what kind of illustration style or style to create for the stamp based on the captured real-life image data.

[0068] <Example of illustration style selection screen> Figure 7 shows an example of the stamp illustration style selection screen displayed in step S405. The screen in Figure 7 displays the stamp illustration styles and accepts selection instructions from the user. For example, the screen in Figure 7 displays the finished stamp image and a brief description of the illustration style for four stamp illustration styles. The screen in Figure 7 also has a user interface in which the user can select and move between illustration styles using the left and right icons at the bottom of the screen, and decide which illustration style to adopt using the "Confirm Icon". The CPU 201 determines the stamp illustration style selected by the user based on the user operation of the operation unit 209.

[0069] Figure 7 shows an example where four illustration styles can be selected, but more illustration styles are possible, and there is no limit to the number of illustration styles as long as they can be implemented through the program operation of the image processing unit 204 or CPU 201. Note that the screen in Figure 7 could also be used as a screen for selecting stamp styles for each SNS app, in a configuration where multiple SNS apps for sales can be selected.

[0070] Furthermore, while the example in Figure 7 shows the finished image of the stamp illustration, it is also possible to display the content of the text that will be superimposed on the generated stamp illustration. For example, it is preferable to display multiple message groups and list them so that the user can easily see them. For example, the first message group is a group of messages related to "greetings" such as "Good morning," "Good night," and "Thank you." The second message group is a group of messages related to "reactions" such as "Yay," "Hmm," "Yes!", and "No." The third message group is a group of messages related to "calls" such as "Attention!", "Please!", and "Are you free now?". The fourth message group is a group of messages related to "emotions or states" such as "Happy," "Sad," "Sleepy," and "Tired."

[0071] Returning to Figure 4, in step S405, the CPU 201 determines the illustration style for the stamp. Subsequently, the CPU 201 repeatedly executes the processes from step S406 onward until it generates a predetermined number of stamp data sheets as defined by the stamp guidelines.

[0072] In step S406, the CPU 201 determines whether it has created the required number of stamp data. If the CPU 201 has created the required number of stamp data (YES in step S406), it proceeds to step S417. If it has not created the required number of stamp data (NO in step S406), it proceeds to step S407.

[0073] In step S407, the CPU 201 displays the screen shown in Figure 8 on the display unit 207, indicating how many stamps remain to be created and a pose guide for the next stamp to be created.

[0074] <Example of remaining number of items screen> Figure 8 shows an example of the screen displayed in step S407. The screen in Figure 8 displays the remaining number of stamps that need to be created for sale and examples of poses for creating stamps. In the example in Figure 8, the screen displays the remaining number of stamps that need to be created and a message prompting the user to take a photo according to the pose of the next stamp to be created.

[0075] On the left side of the screen in Figure 8, the layout of the stamp to be created in the following steps is displayed as an illustration image of a character making an "OK" gesture, and a speech bubble displays text that guides how the subject should be framed.

[0076] On the right side of the screen in Figure 8, an example of a stamp image created by applying the illustration style selected in step S405 is displayed as a finished stamp image, with text information superimposed on it.

[0077] The screen in Figure 8 displays the remaining number of stamps and, before the user proceeds to the next step of actually photographing the subject, provides a visualized image of the pose and guidance. This allows the user to smoothly photograph subjects that possess the "detailed expression, rich facial expressions, and nuances" required for user-friendly stamps.

[0078] Returning to Figure 4, in step S407 described above, the CPU 201 displays the remaining number of stamps, the layout for the next stamp creation, and the subject's shooting pose on the display unit 207. In step S408, the CPU 201, in accordance with the user operation on the operation unit 209, performs guided shooting for the user (subject or photographer) so that shooting is performed according to the pose, and generates image data with the imaging unit 203. Two specific methods of guided shooting will be explained: shooting with voice guidance as shown in Figures 9(a) and (b), and guided shooting using screen displays such as live view as shown in Figures 10(a) and (b).

[0079] <Example of audio guide footage> Figure 9(a) shows a method of taking a photograph using a guide that directly outputs sound from the built-in speaker of the imaging device 200. The tripod 901 supports the imaging device 200. The subject 902 is the subject to be photographed by the imaging device 200. The imaging device 200 outputs sound from its built-in speaker to guide the subject 902 in terms of position and shooting pose.

[0080] Figure 9(b) shows a method of taking images using an imaging device 200 guided by audio output via a wireless earphone 903. A tripod 901 supports the imaging device 200. The subject 902 is the subject to be photographed by the imaging device 200. The wireless earphone 903 is worn on the subject 902's ear. The imaging device 200 transmits audio data wirelessly to the wireless earphone 903 to guide the subject 902 on position and shooting pose. The subject 902 listens to the guide using the wireless earphone 903 worn on the subject 902's ear and is photographed by the imaging device 200.

[0081] In Figures 9(a) and (b), the subject 902 and the photographer operating the imaging device 200 are the same person, and the imaging device 200 is fixed in place using a tripod 901 or the like. Depending on the user's preference, such as ambient noise during shooting and the distance between the imaging device 200 and the subject 902, either the guided shooting method in Figure 9(a) or Figure 9(b) may be used.

[0082] The imaging device 200 operates in a shooting standby state and constantly detects the position and pose of the subject 902 so that the subject is framed within the field of view suitable for stamp generation as shown in step S407 and is in a predetermined pose. The imaging device 200 then guides the subject 902 by giving voice instructions to move up, down, left, right, forward, and backward.

[0083] The detection of subject 902, the region identification of subject 902, and the pose detection of subject 902 may be performed in the image processing unit 204 using known algorithms such as pattern matching, or they may be performed as program processing in the CPU 201. Alternatively, they may be implemented using a method that applies an inference model of deep learning technology.

[0084] Furthermore, once the subject 902 is in the predetermined framing and pose, the imaging device 200 may either automatically release the shutter to take a picture, or notify the user again of the timer countdown before releasing the shutter.

[0085] <Example of shooting guide using live view display> Figures 10(a) and (b) show examples of guided shooting using the live view display of the imaging device 200. In step S407, the imaging device 200 guides the shooting of the subject 902 by displaying an image on the live view panel of the display unit 207, similar to the audio guide in Figure 9, so that the stamp layout and pose shown on the screen in Figure 8 are achieved.

[0086] Figure 10(a) shows an example of live view display guided shooting when the photographer 1001 and the subject 902 are different people. The imaging device 200 displays guide information on the live view panel of the display unit 207 and photographs the subject 902 according to the shooting operation instructions of the photographer 1001. Based on the guide information displayed on the live view panel of the display unit 207, the photographer 1001 verbally instructs the subject 902 on movement and facial expressions so that the layout and pose are suitable for stamp creation. Then, after confirming that the subject 902 is in the predetermined framing and pose, the photographer 1001 performs the shutter release operation as in normal shooting operation and takes a picture with the imaging device 200.

[0087] Figure 10(b) shows an example of live view display guided shooting when the photographer and subject 902 are the same person. The tripod 901 supports the imaging device 200. The imaging device 200 displays guide information on the live view panel of the display unit 207 and takes pictures of subject 902 according to the shooting operation instructions from subject 902 on the remote controller 1002. Subject 902 takes a selfie with the imaging device 200. The display devices such as the live view panel equipped on the imaging device 200 are movable. The display surface of the live view panel is oriented in the same direction as the lens and image sensor, and is rotated so that subject 902 can see the contents of the live view panel. While seeing the contents of the live view panel, subject 902 confirms that subject 902 is in the predetermined framing and pose, and then uses the remote controller 1002 held in front of them to perform a shutter release operation and take a picture with the imaging device 200.

[0088] In addition to the method using the movable live view panel described above, an external monitor or the like may be connected to the imaging device 200 as an attachment, and stamp creation guide information may be displayed on the external monitor or the like in a similar manner.

[0089] Returning to Figure 4, in step S409, the CPU 201 uses the image processing unit 204 to create stamp illustration data in the illustration style selected in step S405, based on the image data generated in step S408. For example, the image processing unit 204 performs a conversion process from a real-life image to an illustration, and further performs a conversion in accordance with the image format specified by the server device 500 and the SNS application operator prior to the pre-check process in the server device 300. For example, when generating stamps to be used in LINE, the image processing unit 204 formats the image as a PNG file with 370 pixels vertically and 320 pixels horizontally, with a transparent background and margins of approximately 10 pixels on all sides. Also, when generating stamps to be used in other SNS applications, the image processing unit 204 converts the image data to conform to the stamp guidelines of those applications.

[0090] In step S410, the CPU 201 previews the stamp illustration data generated in step S409 on the display unit 207, as shown in Figure 11, and asks the user for confirmation whether or not it is OK to send the stamp illustration data to the server device 300.

[0091] <Example of stamp preview display> Figure 11 shows an example of a preview display of stamp illustration data performed in step S410. The example in Figure 11 is an example of converting image data taken with the subject pose shown in Figure 8 in step S407 into stamp illustration data.

[0092] In the center of the screen in Figure 11, the image of the stamp illustration data generated in step S409 is displayed. Below the screen in Figure 11, a user interface is displayed with icons prompting the user to choose whether to proceed to the server device 300's check process using the image as is for the stamp to be sold, or to retake the photo until the user is satisfied with the illustration. The user selects "Proceed" or "Retake" by operating the operation unit 209.

[0093] Returning to Figure 4, in step S411, the CPU 201 determines whether "Proceed" or "Retake" was selected on the screen in Figure 11. If "Proceed" is selected (YES in step S411), the CPU 201 proceeds to step S412. If "Retake" is selected (NO in step S411), the CPU 201 returns to step S407 and performs a retake with the same framing and pose.

[0094] In step S412, the CPU 201 transmits the created stamp illustration data to the server device 300 prior to the sales review (hereinafter referred to as "this review") conducted by the SNS application service provider. The server device 300 performs a conformity check process on the received stamp illustration data based on the flowchart in Figure 5(a) described later, and transmits the pre-check results to the imaging device 200. The conformity check process is a process to check whether the stamp illustration data conforms to the guidelines set by the SNS application provider.

[0095] In step S413, the CPU 201 determines whether it has received a pre-check result from the server device 300 corresponding to the transmission in step S412. If the CPU 201 has not received a pre-check result (NO in step S413), it returns to step S413. If it has received a pre-check result (YES in step S413), it proceeds to step S414.

[0096] If the answer to step S414 is NO, the CPU 201 may, after a predetermined time has elapsed, query the server device 300 for the pre-check results. Alternatively, the CPU 201 may set a timeout value for reception and the number of query retries in advance, and if the pre-check results from the server device 300 cannot be received for any reason, it may perform exception error handling to forcibly terminate the flowchart in Figure 4.

[0097] In step S414, CPU201 determines whether the pre-check result for the stamp illustration data sent in step S412 is OK or NG, based on the pre-check result received in step S413. If the pre-check result is OK (YES in step S414), CPU201 determines that the stamp illustration data is of suitable quality and can proceed to the main review, and proceeds to step S415. If the pre-check result is NG (NO in step S414), CPU201 determines that the stamp illustration data is highly likely to be unsaleable as is, and proceeds to step S416.

[0098] In step S415, CPU201 subtracts the number of stamps remaining needed for sale, and if the remaining number is 1 or more, switches to the template data for the next illustration pose and returns to step S406.

[0099] In step S416, the CPU 201 displays on the display unit 207, for example as shown in Figure 12, that the pre-check result is NG, along with the reason and advice, and then returns to step S407.

[0100] <Example of check result display> Figure 12 shows an example of the pre-check results displayed in step S416. Figure 12 shows a case where the character in the stamp illustration data generated from the subject in the captured image does not fit entirely within the frame, resulting in significant overlap between the text and the character and poor visibility.

[0101] CPU201 displays the reason for the pre-check failure, "Subject not included," along with the stamp illustration data, on display unit 207. Then, CPU201 displays a "Retake" icon below the screen shown in Figure 12, prompting the user to confirm the reason for the failure and proceed to step S407 to take the photo again. When the user selects the "Retake" icon via key selection or touch operation, CPU201 returns to step S407 and performs guided shooting and stamp generation again.

[0102] Returning to Figure 4, step S417 is the process when the pre-check result is OK and the generation of the required number of stamp illustration data for sale has been completed. In step S417, the CPU 201 determines whether or not to request sales review of the stamp illustration data from the server device 500 or the SNS application operator, in response to the user operation on the operation unit 209. Step S417 is the final confirmation step, and the CPU 201 confirms with the user whether or not to request sales review of the image set of stamp illustration data created above. If the CPU 201 requests sales review (YES in step S417), it proceeds to step S418; if the CPU 201 does not request sales review (NO in step S417), it proceeds to step S419.

[0103] In step S418, the CPU 201 sends a request to the server device 300 to act as an agent for the sales review request for the stamp illustration data, and the processing of the flowchart in Figure 4 is completed. The processing that the server device 300 does when it receives the sales review request will be described later using Figure 5(b).

[0104] In step S419, CPU201 executes a cancellation process to discard the stamp illustration data created above, and terminates the process shown in the flowchart in Figure 4.

[0105] <Operation Flow of Server Device 300> Figures 5(a) to 5(c) are flowcharts showing the processing methods of the server device 300. The flowcharts in Figures 5(a) to 5(c) are mainly executed by the CPU 301 of the server device 300 executing a program. Figure 5(a) is a flowchart showing the check process performed when the server device 300 receives stamp illustration data from the imaging device 200. Figure 5(b) is a flowchart showing the proxy sales review request process performed when the server device 300 receives a proxy sales review request for stamp illustration data from the imaging device 200. Figure 5(c) is a flowchart showing the process performed when the server device 300 receives the sales review result from the server device 500.

[0106] <Operation flow when receiving illustration data for stamps> Figure 5(a) is a flowchart showing the processing method when the server device 300 receives illustration data for stamps.

[0107] In step S501, the CPU 301 determines whether or not it has received the stamp illustration data transmitted in step S412 of Figure 4 from the imaging device 200. If the CPU 301 has received the stamp illustration data (YES in step S501), it proceeds to step S502; if it has not received the stamp illustration data (NO in step S501), it returns to step S501.

[0108] In step S502, the CPU 301 determines whether the image format of the received stamp illustration data conforms to the guidelines used by the SNS application. Specifically, the CPU 301 uses the image processing unit 303 to determine whether the received stamp illustration data is in a predetermined file format and is decryptable. If the image format is compliant (YES in step S502), the CPU 301 proceeds to step S503. If the image format is not compliant (NO in step S502), the CPU 301 proceeds to step S510, as there is a high possibility that the data will not be available for sale.

[0109] In step S503, the CPU 301 uses the image processing unit 303 to determine whether the image size of the received stamp illustration data conforms to the guidelines used by the SNS application. If the image size conforms (YES in step S503), the CPU 301 proceeds to step S504. If the image size does not conform (NO in step S503), it proceeds to step S510 because there is a high probability that the image will not be available for sale.

[0110] In step S504, the CPU 301 determines whether or not the stamp illustration data contains a character that illustrates the subject. This determination is made by the CPU 301 using the image processing unit 303 to separate the text portion from the background portion and perform image analysis on the parts other than the text portion. If the stamp illustration data does not contain a character and consists only of text (NO in step S504), the CPU 301 determines that it does not comply with the guidelines and proceeds to step S510. If the stamp illustration data does contain a character (YES in step S504), the CPU 301 proceeds to step S505.

[0111] In step S505, the CPU 301 determines whether the stamp illustration data is too monotonous (too pale) as a design. For example, the CPU 301 uses the image processing unit 303 to calculate the number of colors, a histogram of pixel values, or the dynamic range within the stamp illustration data. The CPU 301 can then determine that the design is too monotonous if the number of colors used is below a predetermined number, or if the dynamic range is localized to a narrow range on the white level side.

[0112] If CPU301 is too monotonous (NO in step S505), it is likely to be unsaleable, so the process proceeds to step S510. If it is not too monotonous (YES in step S505), the process proceeds to step S506.

[0113] In step S506, the CPU 301 determines whether the stamp illustration data has good visibility. While the number of colors, color scheme, and gradation range referenced in step S505 are also indicators related to visibility, in step S506, the CPU 301 determines whether the visibility is good from a broader perspective.

[0114] For example, if the illustration data for a stamp contains multiple illustration characters, includes complex and detailed depictions, or has too much text, it is highly likely to be rejected during the sales review process due to poor readability.

[0115] As an example of a specific method for determining visibility, the CPU 301 uses the image processing unit 303 to detect contours and edges of the stamp illustration data. If more than a predetermined number of edges or contours are detected, it can be determined that the visibility is poor. The method for determining visibility is not limited to any other algorithm that achieves a similar effect.

[0116] If the visibility of the stamp illustration data is poor (NO in step S506), CPU301 proceeds to step S510 because it does not comply with the guidelines. If the visibility of the stamp illustration data is good (YES in step S506), CPU301 proceeds to step S507.

[0117] In step S507, the CPU 301 determines whether the content of the stamp illustration data violates morality and public order and morals. Specifically, content that depicts underage drinking and smoking, sexual expressions, violent expressions, content that incites nationalism, or content that slanders specific individuals or groups will be strictly reviewed according to the guidelines as content that violates morality and public order and morals.

[0118] It is generally difficult to implement the above-described determination using a publicly known rule-based algorithm as program processing in the CPU 301 or as wiring logic in the image processing unit 303. Therefore, in this embodiment, the CPU 301 stores a large amount of image data and evaluation results from the network in a database in advance, and performs the determination using a network model trained with machine learning using the training data and guidance data in that database. This network model is processed at high speed by the machine learning and inference processing unit 308 under a dedicated software or hardware configuration.

[0119] If CPU301 does not violate morals or public order and morals (YES in step S507), it proceeds to step S508. If it does violate morals or public order and morals (NO in step S507), it proceeds to step S510 because there is a risk of violating the guidelines.

[0120] In step S508, the CPU 301 determines whether the content of the stamp illustration data infringes on copyright and portrait rights. Stamp illustration data is generated using a predetermined illustration style, but there are cases where the illustration infringes on copyright or portrait rights if a person or character happens to be included in the image, or if the subject is intentionally disguised.

[0121] As a method for making this determination, the CPU 301 can use a network model that has been trained to determine whether or not the content infringes on copyright or portrait rights, similar to the method used in step S507. Furthermore, the CPU 301 can crawl the internet via the communication unit 304 to determine whether or not there are characters, celebrities, or famous people similar to the stamp illustration data.

[0122] If CPU301 does not infringe on copyright or portrait rights (YES in step S508), it proceeds to step S509. If it does infringe on copyright or portrait rights (NO in step S508), it proceeds to step S510.

[0123] In step S509, the CPU 301 transmits a message to the imaging device 200 indicating that the pre-check results are OK, since all checks in steps S502 to S508 have passed without any problems, and terminates the processing shown in the flowchart in Figure 5(a).

[0124] In step S510, the CPU 301, having received a NO result in any of steps S502 to S508, sends a message to the imaging device 200 indicating that the pre-check result was NG and the reason for it, and terminates the processing shown in the flowchart of Figure 5(a).

[0125] In step S413 of Figure 4, the imaging device 200 receives the transmission content from step S509 or S510.

[0126] As described above, CPU301 determines whether the stamp illustration data conforms to the items defined as guidelines on the messenger app. For example, the items of the guidelines include all or any of the following: image size, data format, image resolution, color mode, data size, background transparency, margins, sexual content, violent content, content with unclear ownership rights, content that violates public order and morals, and content that lacks overall balance.

[0127] <Operation flow when receiving a request for sales review agency services> Figure 5(b) is a flowchart showing the processing method when a request for agency approval of stamp illustration data is received by the server device 300.

[0128] In step S511, the CPU 301 determines whether or not it has received a request to act as an agent for the sales review request for stamp illustration data transmitted from the imaging device 200 in step S418 of Figure 4. In step S418, the imaging device 200 transmits to the server device 300, along with the request to act as an agent for the sales review request, stamp information such as stamp illustration data, account information, bank transfer information, copyright, and sales area necessary for the sales review request. If the CPU 301 has received a request to act as an agent for the sales review request (YES in step S511), it proceeds to step S512; if it has not received a request to act as an agent for the sales review request (NO in step S511), it returns to step S511.

[0129] In step S512, the CPU 301 transmits stamp information, including stamp illustration data, account information, bank transfer information, copyright, and sales area, sent from the imaging device 200, to the server device 500 for the purpose of requesting sales review. After that, the CPU 301 completes the processing shown in the flowchart in Figure 5(b). The server device 500 receives these, performs sales review, and transmits the sales review results to the server device 300.

[0130] <Operation flow when receiving sales review results> Figure 5(c) is a flowchart showing the processing method when the server device 300 receives the sales review results.

[0131] In step S513, the CPU 301 determines whether or not it has received the sales review result of the sales review request sent in step S512 from the server device 500. If the CPU 301 has received the sales review result (YES in step S513), it proceeds to step S514; if it has not received the sales review result (NO in step S513), it returns to step S513.

[0132] In step S514, the CPU 301 saves the stamp illustration data for which sales review was requested, along with the sales review results, to the database 400 connected to the server device 300.

[0133] In step S515, the CPU 301 uses the sales review results stored in the database 400 in step S514 to retrain the suitability check network model in the machine learning and inference processing unit 308.

[0134] <Saving sales review results and retraining the model> Figure 13 shows the retraining of the network model performed in step S515. Stamp illustration data (stamp data) 1301 and sales review results (review results) 1302 are stored in database 400. Stamp illustration data 1301 is the stamp illustration data for which a sales review request has been submitted. Sales review results 1302 are OK or NG. CPU 301 inputs the stamp illustration data 1301 as training data 1303 into network model 1304. Network model 1304 is a network model for determining stamp guideline compliance by machine learning and inference processing unit 308. CPU 301 uses sales review results 1302 as training data 1306 to train network model 1304 so that the training results 1305 of network model 1304 match the training data 1306, which are the correct labels.

[0135] As described above, the CPU 301 performs step S515, and as various users repeatedly submit sales review requests for stamp illustration data, training data 1303 is accumulated in the database 400. As a result, the accuracy of the suitability check network model 1304 is improved.

[0136] Returning to Figure 5(c), in step S516, the CPU 301 sends the sales review results to the mobile terminal 600 owned by the user who created the stamp illustration data, and terminates the processing of the flowchart in Figure 5(c).

[0137] As described above, the image processing system 100 includes an imaging device 200, a server device 300, a database 400, a server device 500, and a mobile terminal 600. In step S409 of Figure 4, the CPU 201 of the imaging device 200 functions as a generation unit and generates stamp data usable in the messenger application by image processing the captured image data. The stamp data is the stamp illustration data described above.

[0138] In steps S502 to S508 of Figure 5(a), the CPU 301 of the server device 300 functions as a determination unit and determines whether the stamp data generated in step S409 conforms to the guidelines. For example, the CPU 301 determines whether the stamp data meets the requirements of image format, image size, presence of a character corresponding to the subject, not being too monotonous, having good visibility, not being contrary to morals and public order and morals, or not infringing on copyright and portrait rights.

[0139] In steps S507 and S508, the CPU 301 can make a determination using the network model 1304 in Figure 13. In step S515 in Figure 5(c), the CPU 301 functions as a learning unit and learns the network model 1304 based on the stamp data 1301 and the sales review request review results 1302.

[0140] In step S416 of Figure 4, the CPU 201 functions as a display control unit and controls the system to display the judgment results from steps S502 to S508, as shown in Figure 12. Furthermore, if the CPU 201 determines in steps S502 to S508 that the stamp data does not conform to the guidelines, as shown in Figure 12, it controls the system to display the reason for the non-conformity. Also, if the CPU 201 determines in steps S502 to S508 that the stamp data does not conform to the guidelines, as shown in Figure 12, it controls the system to display a prompt asking whether to retake the photograph.

[0141] In step S403, the CPU 201 displays the contents of the contract and prompts the user to agree to the contents of the contract, as shown in Figures 6(a) to (c). In step S404, if the user indicates that they agree to the contents of the contract, the CPU 201 proceeds to step S405.

[0142] In step S405, the CPU201 functions as a selection unit and, as shown in Figure 7, selects one of several stamp data styles in response to user input.

[0143] In step S407, the CPU 201 controls the system to display example poses to guide the subject's pose, as shown in Figure 8, before taking the picture in step S408. The CPU 201 also controls the system to display example stamp data that would be generated if the picture were taken using the guided pose, as shown in Figure 8.

[0144] In step S408, the CPU201 functions as the shooting unit and generates image data by shooting the subject. The CPU201 detects the position of the subject within the shooting angle of view, guides the camera to move to the subject's position, and prompts the camera to press the shutter release button or performs an automatic shutter release when the subject reaches the guided position. For example, as shown in Figure 9(a) or (b), the CPU201 guides the camera to move to the subject's position using voice. Alternatively, as shown in Figure 10(a) or (b), the CPU201 guides the camera to move to the subject's position by overlaying graphics onto the live view.

[0145] In step S409, the CPU 201 generates stamp data by processing the image data captured in step S408, according to the stamp data style selected in step S405.

[0146] In step S410, the CPU 201 displays the stamp data generated in step S409, as shown in Figure 11, and controls the system to display a query asking whether it is OK to proceed with processing the stamp data. In step S411, if the CPU 201 is instructed to proceed with processing the stamp data, it proceeds to step S412.

[0147] The loop processing in steps S407 to S415 is performed multiple times. In step S406, if the server device 300 determines that a predetermined number of stamp data conform to the guidelines, the CPU 201 proceeds to step S417. In step S418, the CPU 201 sends a request to the server device 300 to submit a request for sales review of a predetermined number of stamp data to the messenger app operating website. In step S512 of Figure 5(b), upon receiving the request, the CPU 301 functions as a requesting unit and submits a request for sales review of a predetermined number of stamp data to the messenger app operating website.

[0148] In step 407, the CPU 201 controls the system to display the number of remaining stamp data required to reach a determination that a predetermined number of stamp data conform to the guidelines, as shown in Figure 8.

[0149] Next, the processing method of the imaging device 200 in Figure 4 will be explained. In step S412, the CPU 201 functions as a transmitter and sends stamp data usable in the messenger application, generated based on the captured image data, to the server device 300. In step S413, the CPU 201 receives a determination result from the server device 300 indicating whether the stamp data usable in the messenger application, generated based on the captured image data, conforms to the guidelines. In step S416, the CPU 201 is controlled to display the determination result.

[0150] Next, the processing method of the server device 300 in Figure 5(a) will be explained. In step S501, the CPU 301 functions as a receiver and receives stamp data usable in the messenger application, which is generated based on the captured image data. In steps S502 to S508, the CPU 301 determines whether the stamp data usable in the messenger application, which is generated based on the captured image data, conforms to the guidelines.

[0151] As described above, according to this embodiment, an individual user does not need to create illustration stamp data from scratch. The user can easily create original stamp data based on captured image data, register it on a sales site, and sell it. The image processing system 100 can reduce the effort required of the user to generate stamp data that conforms to the guidelines.

[0152] (Second embodiment) The first embodiment shows a method in which the imaging device 200 converts real-world images captured in shooting mode according to a guide into stamp illustration data. The second embodiment shows a method in which the imaging device 200 transmits already captured real-world image data to the server device 300 in playback mode, and the server device 300 converts the received image data into stamp illustration data.

[0153] The second embodiment will be described below with reference to Figures 14 and 15. In the description of the second embodiment, parts that are common to the first embodiment described above will be omitted as appropriate.

[0154] In addition to the decryption and analysis processing of stamp illustration data described in the first embodiment above, the image processing unit 303 in Figure 3 also has the function of converting the captured image data held by the image processing unit 204 in Figure 2 into stamp illustration data.

[0155] <Operation flow of imaging device 200> Figure 14 is a flowchart showing the processing method of the imaging device 200 according to the second embodiment.

[0156] In step S1401, the CPU 201 is triggered by a mode setting from the operation unit 209, which has an interface such as buttons, a directional pad, or a touch panel, and determines whether the operating mode of the imaging device 200 is playback mode. If the operating mode is not playback mode (NO in step S1401), the CPU 201 terminates the processing of the flowchart in Figure 14. If the operating mode is playback mode (YES in step S1401), the process proceeds to step S1402.

[0157] In step S1402, CPU 201 performs the same processing as in step S402 in Figure 4. If stamp creation mode is not selected (NO in step S1402), CPU 201 returns to step S1401; if stamp creation mode is selected (YES in step S1402), it proceeds to step S1403.

[0158] In step S1403, CPU201 performs the same processing as in step S403 in Figure 4.

[0159] In step S1404, CPU201 performs the same processing as in step S404 in Figure 4. If the user does not agree to the service (NO in step S1404), CPU201 returns to step S1401. If the user agrees to the service (YES in step S1404), it proceeds to step S1405.

[0160] In step S1405, the CPU 201 downloads illustration styles from the server device 300 to update the repertoire of selectable styles, as the server device 300 creates illustration data for stamps. After that, the CPU 201 performs the same process as in step S405 in Figure 4.

[0161] In step S1406, CPU 201 performs the same processing as in step S406 in Figure 4. If CPU 201 has created the required number of stamp data (YES in step S1406), it proceeds to step S1415. If it has not created the required number of stamp data (NO in step S1406), it proceeds to step S1407.

[0162] In step S1407, CPU 201 performs the same processing as in step S407 in Figure 4.

[0163] In step S1408, the CPU 201 searches for captured image data saved by the user in the recording / playback unit 206 in response to user operation, and selects actual image data that captures a layout and subject pose suitable for stamp creation as instructed in step S1407. The actual image data is the original image data.

[0164] In step S1409, the CPU 201 sends the live-action image data selected in step S1408 and the illustration style selected in step S1405 to the server device 300 in order to convert them into illustration data for stamps.

[0165] In step S1410, the CPU 201 determines whether it has received stamp illustration data and pre-check results for stamp illustration data corresponding to the real-life image data transmitted in step S1409 from the server device 300. If the CPU 201 has received the stamp illustration data and pre-check results (YES in step S1410), it proceeds to step S1411. If the CPU 201 has not received the stamp illustration data and pre-check results (NO in step S1410), it returns to step S1410. If the answer to step S1410 is NO, the CPU 201 can inquire about the receipt of stamp illustration data and pre-check results from the server device 300 after a predetermined time has elapsed.

[0166] In step S1411, the CPU 201 displays a preview of the stamp illustration data created by the server device 300 and the pre-check results received from the server device 300 on the display unit 207. For example, the CPU 201 displays a screen on the display unit 207 that combines the preview of the stamp illustration data shown in Figure 11 and the pre-check results shown in Figure 12.

[0167] In step S1412, CPU 201 performs the same process as in step S414 in Figure 4. If the pre-check result is OK (YES in step S1412), CPU 201 proceeds to step S1413; if the pre-check result is NG (NO in step S1412), it proceeds to step S1414.

[0168] In step S1413, CPU 201 performs the same process as in step S415 in Figure 4, and then returns to step S1406.

[0169] In step S1414, since the pre-check result is NG, the CPU 201 automatically searches the recorded image data stored in the recording / playback unit 206, displays image candidates similar to the image selected in step S1408 on the display unit 207, and returns to step S1407.

[0170] As an example of a method for determining similar images, CPU201 can determine whether an image is similar or not based on known image similarity determination algorithms such as pattern matching, or on metadata such as the date and location of the image. The method for determining similar images is not limited to this.

[0171] In step S1415, CPU 201 performs the same processing as in step S417 in Figure 4. If CPU 201 requests a sales review (YES in step S1415), it proceeds to step S1416; if it does not request a sales review (NO in step S1415), it proceeds to step S1417.

[0172] In step S1416, the CPU 201 performs the same process as in step S418 in Figure 4, and terminates the process shown in the flowchart in Figure 14.

[0173] In step S1417, the CPU 201 performs the same process as in step S419 in Figure 4, and terminates the process shown in the flowchart in Figure 14.

[0174] <Operation Flow of Server Device 300> Figure 15 is a flowchart showing the processing method of the server device 300 according to the second embodiment.

[0175] In step S1501, the CPU 301 determines whether it has received the real-world image data transmitted in step S1409 in Figure 14 and the illustration style selected in step S1405 in Figure 14 from the imaging device 200. If the CPU 301 has received the real-world image data and illustration style (YES in step S1501), it proceeds to step S1502 to convert the real-world image data into stamp illustration data. If the CPU 301 has not received the real-world image data and illustration style (NO in step S1501), it returns to step S1501.

[0176] In step S1502, the CPU 301 performs the same processing as in step S409 in Figure 4. Based on the real-world image data received in step S1501, the CPU 301 uses the image processing unit 204 to create stamp illustration data using the illustration style received in step S1501.

[0177] In step S1503, the CPU 301 performs the same processing on the stamp illustration data created in step S1502 as in step S502 in Figure 5(a). If the image format is compliant (YES in step S1503), the CPU 301 proceeds to step S1504; otherwise, it proceeds to step S1511.

[0178] In step S1504, the CPU 301 performs the same processing on the stamp illustration data created in step S1502 as in step S503 in Figure 5(a). If the image size is compliant (YES in step S1504), the CPU 301 proceeds to step S1505; otherwise, it proceeds to step S1511.

[0179] In step S1505, the CPU 301 performs the same processing on the stamp illustration data created in step S1502 as in step S504 in Figure 5(a). If the character is not present (NO in step S1505), the CPU 301 proceeds to step S1511; if a character is present (YES in step S1505), it proceeds to step S1506.

[0180] In step S1506, the CPU 301 performs the same processing on the stamp illustration data created in step S1502 as in step S505 in Figure 5(a). If the data is too monotonous (NO in step S1506), the CPU 301 proceeds to step S1511; otherwise, it proceeds to step S1507.

[0181] In step S1507, the CPU 301 performs the same processing on the stamp illustration data created in step S1502 as in step S506 in Figure 5(a). If the stamp illustration data is not easily visible (NO in step S1507), the CPU 301 proceeds to step S1511. If the stamp illustration data is easily visible (YES in step S1507), the CPU 301 proceeds to step S1508.

[0182] In step S1508, the CPU 301 performs the same processing on the stamp illustration data created in step S1502 as in step S507 in Figure 5(a). If the CPU 301 determines that the data does not violate morals or public order and morals (YES in step S1508), it proceeds to step S1509; otherwise, it proceeds to step S1511.

[0183] In step S1509, the CPU 301 performs the same processing on the stamp illustration data created in step S1502 as in step S508 in Figure 5(a). If the CPU 301 does not infringe on copyright or portrait rights (YES in step S1509), it proceeds to step S1510; if it does infringe on copyright or portrait rights (NO in step S1509), it proceeds to step S1511.

[0184] In step S1510, the CPU 301 transmits to the imaging device 200 that the stamp illustration data and the pre-check results are OK, similar to step S509 in Figure 5(a), and terminates the processing of the flowchart in Figure 15.

[0185] In step S1511, the CPU 301, similar to step S510 in Figure 5(a), sends the stamp illustration data and a message to the imaging device 200 indicating that the pre-check result is NG and the reason for this, thereby ending the process shown in the flowchart of Figure 15.

[0186] As described above, in step S1502 of Figure 15, the CPU 301 of the server device 300 functions as a generation unit and generates stamp data usable in the messenger application by image processing the captured image data. The stamp data is the stamp illustration data described above. In steps S1503 to S1509 of Figure 15, the CPU 301 of the server device 300 functions as a determination unit and determines whether or not the stamp data generated in step S1502 conforms to the guidelines.

[0187] In step S1408 of Figure 14, the CPU 201 functions as a selection unit and selects one of several captured image data in response to user operation. In step S1502 of Figure 15, the CPU 301 generates stamp data by image processing the captured image data selected in step S1408.

[0188] In step S1412 of Figure 14, if the server device 300 determines that the stamp data does not conform to the guidelines, the CPU 201 proceeds to step S1414. In step S1414, the CPU 201 searches for image data similar to the image data from which the stamp data was generated among multiple captured image data, and controls the system to display the similar image data. Then, in step S1408, the CPU 201 selects one of the similar image data in response to user input.

[0189] Next, the processing method of the imaging device 200 in Figure 14 will be explained. In step S1409, the CPU 201 transmits the captured image data to the server device 300. In step S1410, the CPU 201 receives from the server device 300 the result of a determination of whether the stamp data usable in the messenger application, which was generated based on the captured image data, conforms to the guidelines. In step S1411, the CPU 201 is controlled to display the determination result.

[0190] Next, the processing method of the server device 300 in Figure 15 will be explained. In step S1501, the CPU 301 receives the captured image data. In step S1502, the CPU 301 generates stamp data that can be used in the messenger application based on the captured image data. In steps S1503 to S1509, the CPU 301 determines whether the generated stamp data conforms to the guidelines.

[0191] As described above, according to this embodiment, the server device 300 can generate stamp data based on the captured image data.

[0192] (Third embodiment) The first and second embodiments described above show a method in which the server device 300 performs a pre-check of stamp illustration data generated from image data captured by the user. In the third embodiment, the pre-check process for stamp illustration data is performed by the server device 500, not the server device 300.

[0193] In the first and second embodiments, server device 300 and server device 500 were described as separate devices. In this embodiment, server device 300 and server device 500 are configured as a single server device. Server device 500 has a configuration similar to the hardware configuration in Figure 2 and possesses both the functions of server device 300 and server device 500 in the first or second embodiment. This reduces network communication data traffic between server device 300 and server device 500, resulting in a shorter turnaround time for users who wish to sell stamps, from taking a photo to receiving the sales review results.

[0194] (Fourth embodiment) In the first to third embodiments described above, the user uses the imaging device 200 to perform a shooting operation to create illustration data for stamps, and receives the final sales review results on a mobile terminal 600 owned by the user. The specific product form of the mobile terminal 600 is a smartphone or a tablet. In recent years, mobile terminals 600 have been equipped with image sensors that have the same level of performance as digital cameras and video cameras.

[0195] Therefore, in the fourth embodiment, the mobile terminal 600 may be integrated with the imaging device 200, having the same or similar functional block configuration as the imaging device 200. The mobile terminal 600 can perform the same processing as the imaging device 200 described in the first or second embodiment above. As a result, the mobile terminal 600 has the same processing functions as the imaging device 200, and can complete a series of guided shooting, automatic creation of stamp illustration data, and processing from pre-checking the stamp illustration data to receiving the sales review results within a single device.

[0196] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0197] Furthermore, the embodiments described above are merely examples illustrating how to implement this disclosure, and they should not be interpreted as limiting the technical scope of this disclosure. In other words, this disclosure can be implemented in various ways without departing from its technical concept or its main features.

[0198] This embodiment includes the following configurations, methods, and programs. (Composition 1) A generation method for generating stamp data usable in a messenger app by processing captured image data, A determination means for determining whether the stamp data generated by the generation means conforms to the guidelines. An image processing system characterized by having the following features. (Configuration 2) The aforementioned image processing system comprises an imaging device and a server device. The imaging device has the generation means, The image processing system according to configuration 1, characterized in that the server device has the determination means. (Composition 3) The aforementioned image processing system includes a server device, The image processing system according to configuration 1, characterized in that the server device has the generation means and the determination means. (Composition 4) The image processing system according to any one of configurations 1 to 3, further comprising a display control means for controlling the display of the determination result of the determination means. (Composition 5) The image processing system according to configuration 4, characterized in that the display control means controls the display to show the reason for the non-compliance when the determination means determines that the stamp data does not conform to the guidelines. (Composition 6) The image processing system according to any one of configurations 1 to 5, characterized in that the determination means determines whether the stamp data generated by the generation means satisfies the requirements of image format, image size, presence of a character corresponding to the subject, not being too monotonous, having good visibility, not being contrary to morals and public order and morals, or not infringing on copyright and portrait rights. (Composition 7) The system further includes a means for generating image data by photographing a subject, The image processing system according to any one of configurations 1 to 6, characterized in that the generation means generates the stamp data by image processing the image data captured by the shooting means. (Composition 8) The system further includes a display control means that controls the determination means to display the determination result, The image processing system according to configuration 7, characterized in that the display control means controls to display an inquiry asking whether or not to retake the photo if the determination means determines that the stamp data does not conform to the guidelines. (Composition 9) The image processing system according to configuration 7, further comprising a display control means that controls the display of pose examples to guide the subject's pose before taking a photograph using the aforementioned photographic means. (Composition 10) The image processing system according to configuration 9, characterized in that the display control means controls to display an example of stamp data generated when a photograph is taken using the guided pose example. (Composition 11) The image processing system according to any one of configurations 7 to 10, characterized in that the shooting means detects the position of a subject within the field of view to be photographed, guides the subject to move to the guided position, and prompts the user to press the shutter release button or performs an automatic shutter release operation when the subject reaches the guided position. (Composition 12) The image processing system according to configuration 11, characterized in that the shooting means guides the position of the subject by sound. (Composition 13) The image processing system according to configuration 11, characterized in that the shooting means guides the position of the subject by superimposing graphics onto the live view. (Composition 14) The system further includes a first selection means that selects one of several stamp data styles in response to user input. The image processing system according to any one of configurations 1 to 13, characterized in that the generation means generates the stamp data by image processing the captured image data according to the stamp data style selected by the first selection means. (Composition 15) The generation by the generation means and the determination by the determination means are performed multiple times. The image processing system according to any one of configurations 1 to 14, further comprising a means for requesting a sales review of the predetermined number of stamp data from the messenger app operating website if the determination means determines that a predetermined number of stamp data conforms to the guidelines. (Composition 16) The image processing system according to configuration 15, further comprising a display control means for controlling the display of the number of remaining stamp data required to reach a determination that the predetermined number of stamp data conforms to the guidelines. (Composition 17) The system further includes a display control means that displays the stamp data generated by the generation means and controls the system to display an inquiry asking whether or not to proceed with processing the stamp data. The image processing system according to any one of configurations 1 to 16, characterized in that the determination means performs the determination when it is instructed that it is OK to proceed with processing the stamp data. (Composition 18) The system further includes a display control means that controls the display to show the contents of the contract and to display an inquiry asking whether or not the user agrees to the contents of the contract. The image processing system according to any one of configurations 1 to 17, characterized in that, if an instruction is given to agree to the contents of the aforementioned contract, the generation by the generation means and the determination by the determination means are performed. (Composition 19) The determination means performs the determination using a network model, The image processing system according to configuration 15, further comprising a learning means for training the network model based on the stamp data and the results of the sales review request. (Composition 20) It further includes a second selection means that selects one of several captured image data in response to user operation, The image processing system according to any one of configurations 1 to 6, characterized in that the generation means generates the stamp data by image processing the captured image data selected by the second selection means. (Composition 21) If the determination means determines that the stamp data does not conform to the guidelines, the system will search among the multiple captured image data for image data similar to the image data from which the stamp data was generated, and control the system to display the similar image data. The image processing system according to configuration 20, characterized in that the second selection means selects one of the similar image data in response to user operation. (Composition 22) A receiving means for receiving captured image data, or stamp data usable in a messenger app generated based on captured image data, A determination means for determining whether stamp data usable in a messenger app, generated based on the captured image data, conforms to the guidelines. A server device characterized by having the following features. (Composition 23) A transmission means for sending captured image data, or stamp data usable in a messenger app generated based on captured image data, to a server device, A receiving means that receives from the server device a determination result of whether or not the stamp data usable in the messenger app, which was generated based on the captured image data, conforms to the guidelines. A display control means that controls the display of the aforementioned determination result. An imaging device characterized by having the following features. (Method 1) The process involves generating stamp data usable in a messenger app by processing the captured image data, and A determination step to determine whether the stamp data generated by the generation step conforms to the guidelines. A processing method for an image processing system, characterized by having the following features. (Method 2) A receiving step that receives captured image data, or stamp data usable in a messenger app generated based on captured image data, A determination step to determine whether the stamp data usable in the messenger app, generated based on the captured image data, conforms to the guidelines. A processing method for a server device, characterized by having the following features. (Method 3) A transmission step of sending captured image data, or stamp data usable in a messenger app generated based on captured image data, to a server device, A receiving step in which the server device receives a result of a determination of whether or not the stamp data usable in the messenger app, which was generated based on the captured image data, conforms to the guidelines. A display control step that controls the display of the aforementioned determination result. A processing method for an imaging device, characterized by having the following features. (Program 1) A program for causing a computer to function as one of the image processing systems, server devices, or imaging devices described in any one of configurations 1 to 23. [Explanation of Symbols]

[0199] 100 Image processing system, 200 Imaging device, 300 Server device, 400 Database, 500 Server device, 600 Mobile terminal, 201 CPU, 202 Primary memory unit, 203 Imaging unit, 204 Image processing unit, 205 Communication unit, 206 Recording / playback unit, 207 Display unit, 208 Audio processing unit, 209 Operation unit, 210 CPU bus, 211 Work memory, 212 Memory bus, 301 CPU, 302 Primary memory unit, 303 Image processing unit, 304 Communication unit, 305 Recording / playback unit, 306 Display unit, 307 Operation unit, 308 Machine learning and inference processing unit, 309 CPU bus, 310 Work memory, 311 Memory bus

Claims

1. A generation method for generating stamp data usable in a messenger app by processing captured image data, A determination means for determining whether the stamp data generated by the generation means conforms to the guidelines, When the number of stamp data determined to conform to the guidelines by the determination means reaches a predetermined number required for requesting sales review from the messenger app operating website, the request means makes a request for sales review from the messenger app operating website for the predetermined number of stamp data determined to conform to the guidelines. The display control means controls the display to show the number of remaining stamp data required until the number of stamp data determined by the determination means to conform to the guidelines reaches the predetermined number, The generation means generates a plurality of stamp data, The determination means determines whether each of the plurality of stamp data generated by the generation means conforms to the guidelines. An image processing system characterized by the following:

2. The aforementioned image processing system comprises an imaging device and a server device. The imaging device has the generation means, The image processing system according to claim 1, characterized in that the server device has the determination means.

3. The aforementioned image processing system includes a server device, The image processing system according to claim 1, characterized in that the server device has the generation means and the determination means.

4. The image processing system according to claim 1, further comprising a display control means for controlling the display of the determination result of the determination means.

5. The image processing system according to claim 4, characterized in that the display control means controls the display to show the reason for the non-compliance when the determination means determines that the stamp data does not conform to the guidelines.

6. The image processing system according to claim 1, characterized in that the determination means determines whether the stamp data generated by the generation means satisfies the requirements of image format, image size, presence of a character corresponding to the subject, not being too monotonous, having good visibility, not being contrary to morals and public order and morals, or not infringing on copyright and portrait rights.

7. The system further includes a means for generating image data by photographing a subject, The image processing system according to claim 1, characterized in that the generation means generates the stamp data by image processing the image data captured by the shooting means.

8. The system further includes a display control means that controls the determination means to display the determination result, The image processing system according to claim 7, characterized in that the display control means is controlled to display an inquiry asking whether or not to take a new photograph if the determination means determines that the stamp data does not conform to the guidelines.

9. The image processing system according to claim 7, further comprising a display control means that controls the display of pose examples to guide the subject's pose before taking a photograph using the aforementioned photographic means.

10. The image processing system according to claim 9, characterized in that the display control means controls to display an example of stamp data generated when a photograph is taken using the guided pose example.

11. The image processing system according to claim 7, characterized in that the shooting means detects the position of a subject within the field of view to be photographed, guides the subject to move to the guided position, and prompts the user to press the shutter release button or performs an automatic shutter release operation when the subject reaches the guided position.

12. The image processing system according to claim 11, characterized in that the shooting means guides the position of the subject by sound.

13. The image processing system according to claim 11, characterized in that the shooting means guides the position of the subject by superimposing graphics onto the live view.

14. The system further includes a first selection means that selects one of several stamp data styles in response to user input. The image processing system according to claim 1, characterized in that the generation means generates the stamp data by image processing the captured image data according to the stamp data style selected by the first selection means.

15. The system further includes a display control means that displays the stamp data generated by the generation means and controls the system to display an inquiry asking whether or not to proceed with processing the stamp data. The image processing system according to claim 1, characterized in that the determination means performs the determination when it is instructed that it is OK to proceed with processing the stamp data.

16. The system further includes a display control means that controls the display to show the contents of the contract and to display an inquiry asking whether or not the user agrees to the contents of the contract. The image processing system according to claim 1, characterized in that, if an instruction is given to agree to the contents of the aforementioned contract, the generation by the generation means and the determination by the determination means are performed.

17. The determination means performs the determination using a network model, The image processing system according to claim 1, further comprising a learning means for learning the network model based on the stamp data and the results of the sales review request.

18. The system further includes a second selection means that, in response to user input, selects one of several pre-captured image data. The image processing system according to claim 1, characterized in that the generation means generates the stamp data by image processing the captured image data selected by the second selection means.

19. If the determination means determines that the stamp data does not conform to the guidelines, the system will search among the multiple captured image data for image data similar to the image data from which the stamp data was generated, and control the system to display the similar image data. The image processing system according to claim 18, characterized in that the second selection means selects one of the similar image data in response to user operation.

20. The process involves generating stamp data usable in a messenger app by processing the captured image data, and A determination step to determine whether the stamp data generated by the generation step conforms to the guidelines, If the number of stamp data determined to conform to the guidelines in the judgment step reaches a predetermined number required for requesting sales review from the messenger app operating website, the request step is to request sales review from the messenger app operating website for the predetermined number of stamp data determined to conform to the guidelines, The display control step includes controlling the display to show the number of remaining stamp data required until the number of stamp data determined to conform to the guidelines by the determination step reaches the predetermined number, In the generation step, multiple stamp data are generated, In the determination step, it is determined whether each of the plurality of stamp data generated in the generation step conforms to the guidelines. A processing method for an image processing system characterized by the following.

21. A program for causing a computer to function as one of the means of an image processing system described in any one of claims 1 to 19.

Images