Information processing methods, information processing systems, and programs

Abstract

This invention provides an information processing method, information processing system, and program that allow for adjustment of changes in motion represented in a generated sequence of images. [Solution] In the information processing apparatus, the control unit includes an acquisition unit that acquires first image data and state information. The state information is at least one of a second image data showing the image after a change in the first image data and a prompt showing the state after a change in the first image data. The information processing apparatus also includes a generation unit that generates a plurality of in-between image data which are image data that interpolates between the first image data and the state information, and generates a first image sequence which is data arranged in chronological order by the first image data, the plurality of in-between image data and the state information; an adjustment unit that adjusts the change in motion expressed in the first image sequence based on the first image sequence and input adjustment information; and an output unit that outputs a second image sequence which is the adjusted image sequence.

Description

Technical Field

[0004] , , , , , , , , , , , ,

[0006] , , , ,

[0005] , , , , , ,

[0007] , ,

[0003] , ,

[0001] The present invention relates to an information processing method, an information processing system, and a program.

Background Art

[0002] Patent Document 1 discloses an image generation device.

[0003] The image generation device of Patent Document 1 is trained to output a generated image at a generation time that is not the reference time using time series information including a combination of a learning image at each reference time and learning structure information indicating the structure of the learning image. This image generator is trained to receive feedback from an image identifier trained to identify whether the input image is a learning image or a generated image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] In the method described in one embodiment above, a first image sequence is obtained in which the first image data and the state information (at least one of the second image data and the prompt) are interpolated with a plurality of intermediate image data, thereby changing at a constant velocity between the first image data and the state information. Subsequently, the first image sequence is adjusted using input adjustment information to impart an accelerated change in motion. Since no special equipment is required to impart this accelerated change in motion, the configuration is simpler.

[0009] Therefore, the method of this embodiment can improve the functionality of a computer to achieve at least one of the following (1) to (4): (1) The computer's processing speed can be increased. (2) The computer's power consumption can be reduced. (3) The computer's communication speed can be increased. (4) The resources saved in the computer can be used for other core functions.

[0010] According to this embodiment, it is possible to adjust the changes in motion in the generated image sequence. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram showing the configuration of information processing system 100. [Figure 2] This is a block diagram showing the hardware configuration of the information processing device 200. [Figure 3] This is a block diagram showing the hardware configuration of terminal 300. [Figure 4] This is a block diagram showing the functions realized by the information processing device 200 (control unit 210). [Figure 5] This is an activity diagram showing the flow of the information processing method executed by the information processing device 200. [Figure 6] This is an activity diagram showing the flow of the information processing method executed by the information processing device 200. [Figure 7] This is an activity diagram showing the flow of the information processing method executed by the information processing device 200. [Figure 8] This figure shows an example of a graph editor used when entering adjustment information. [Figure 9] This diagram shows the process of generating a second image sequence after adjusting for changes in motion, using the first and second image data. [Figure 10] This diagram supplements the processing of Activity A280. [Figure 11] This diagram supplements the processing of activities A310 through A340. [Figure 12] This diagram supplements the processing of activities A310 through A340. [Figure 13] This diagram supplements the processing of activities A310 through A340. [Figure 14] This diagram supplements the processing of activities A310 through A340. [Figure 15] This diagram supplements the processing of Activity A430. [Figure 16] This figure shows a comparison between image data that has undergone color reduction processing using the method of this embodiment and image data that has undergone color reduction processing using a conventional method. [Figure 17] This is a diagram showing a comparison between image data obtained by performing color reduction processing using the method of this embodiment and image data obtained by performing color reduction processing using a conventional method. [Figure 18] This is a diagram for explaining a prompt which is an example of state information. [Figure 19] This is a diagram showing an example of settings for a line graph in a graph editor.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the embodiments described below can be combined with each other.

[0013] Incidentally, a program for realizing software appearing in one embodiment may be provided as a non-transitory computer-readable medium that can be read by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its function is realized on a client terminal (so-called cloud computing).

[0014] Also, in various information processes according to one embodiment, an input and an output corresponding to the input can be realized. Here, if an output is obtained as a result of the input, the mode of information (hereinafter referred to as reference information) referred to in such information processing is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, a predetermined function (including a determination formula such as a regression formula constructed by a statistical method), a learned model in which the correlation between an input and an output has been learned in advance, or a large language model capable of outputting a desired result by inputting a prompt.

[0015] Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense.

[0016] Furthermore, a circuit in a broad sense is a circuit realized by combining at least a suitable combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, it includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0017] 1. Hardware Configuration Section 1 describes the hardware configuration of this embodiment.

[0018] 1-1. Information Processing System 100 Figure 1 is a diagram illustrating the configuration of the information processing system 100. The information processing system 100 comprises an information processing device 200 and a terminal 300, which are connected via a network. These components will be further explained. Here, the system exemplified in the information processing system 100 consists of one or more devices or components. Therefore, for example, even the information processing device 200 alone can be a system exemplified in the information processing system 100.

[0019] 1-2. Information processing device 200 Figure 2 is a block diagram showing the hardware configuration of the information processing device 200. The information processing device 200 includes a control unit 210, a storage unit 220, and a communication unit 250, and these components are electrically connected within the information processing device 200 via a communication bus 260. The information processing device 200 is, for example, a server. Each component will be described further.

[0020] The control unit 210 performs processing and control of the overall operation related to the information processing device 200. The control unit 210 is, for example, a Central Processing Unit (CPU) (not shown). The control unit 210 realizes various functions related to the information processing device 200 by reading predetermined programs stored in the storage unit 220. That is, information processing by software stored in the storage unit 220 is concretely realized by the control unit 210, which is an example of hardware, and can be executed as each functional unit included in the control unit 210. These will be explained further in Section 2. Note that the control unit 210 is not limited to being a single unit, and may be implemented with multiple control units 210 for each function, or a combination thereof.

[0021] The storage unit 220 stores various information necessary for information processing by the information processing device 200. This can be done, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 200 executed by the control unit 210, or as memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. A combination of these may also be used.

[0022] The communication unit 250 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as 5G / LTE / 3G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the information processing device 200 communicates various information with the terminal 300 via the network through the communication unit 250.

[0023] 1-3. Terminal 300 Figure 3 is a block diagram showing the hardware configuration of terminal 300. Terminal 300 includes a control unit 310, a storage unit 320, a display unit 330, an input unit 340, and a communication unit 350, and these components are electrically connected within terminal 300 via a communication bus 360. Terminal 300 is, for example, a desktop personal computer. The explanation of the control unit 310, storage unit 320, and communication unit 350 is substantially the same as the explanation of the control unit 210, storage unit 220, and communication unit 250 in the information processing device 200, so it is omitted here.

[0024] The display unit 330 may be included in the casing of the terminal 300 or it may be an external component. The display unit 330 displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably done by using a display device such as a CRT display, liquid crystal display, organic EL display, or plasma display, depending on the type of terminal 300. In the following description, the display unit 330 will be described as being included in the casing of the terminal 300.

[0025] The input unit 340 may be included in the casing of the terminal 300 or it may be an external component. For example, the input unit 340 may be integrated with the display unit 330 and implemented as a touch panel. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, a switch button, mouse, QWERT keyboard, etc. may be used instead of a touch panel. In other words, the input unit 340 receives operation input made by the user. This input is transmitted as a command signal to the control unit 310 via the communication bus 360. The control unit 310 can then perform predetermined controls and calculations as needed.

[0026] 2. Functional Configuration Section 2 will describe the functional configuration of this embodiment. As mentioned above, the information processing by the software stored in the memory unit 220 is specifically realized by the control unit 210, which is an example of hardware, and can be executed as each functional unit included in the control unit 210.

[0027] Figure 4 is a block diagram showing the functions realized by the information processing device 200 (control unit 210). As described above, the information processing device 200 (information processing system 100) includes a control unit 210. Specifically, the information processing device 200 (control unit 210) is configured to execute each step in the information processing method of this embodiment. The information processing device 200 (control unit 210) includes an acquisition unit 211, a generation unit 212, an adjustment unit 213, an output unit 214, a deletion unit 215, a separation unit 216, a color reduction unit 217, and an integration unit 218, corresponding to each step in the information processing method of this embodiment. The following will describe each unit in relation to each step.

[0028] The acquisition unit 211 is configured to acquire various types of information. The acquisition unit 211 is configured to perform acquisition steps. For example, the acquisition unit 211 acquires first image data and state information. The first image data is a reference image. Here, the state information is at least one of a second image data showing the image after the first image data has been changed, and a prompt showing the state after the first image data has been changed. The acquisition unit 211 also acquires a third image data, which is a colored image data.

[0029] The generation unit 212 is configured to generate various types of information. The generation unit 212 is configured to execute generation steps. For example, the generation unit 212 generates multiple in-between image data, which are image data that interpolates between the first image data and the state information. The generation unit 212 also generates a first image sequence, which is data in which the first image data, the multiple in-between image data, and the state information are arranged in chronological order.

[0030] The adjustment unit 213 is configured to adjust various types of information. The adjustment unit 213 is configured to perform adjustment steps. For example, the adjustment unit 213 adjusts the changes in motion represented in the first image sequence based on the first image sequence and input adjustment information. Here, the adjustment information is information used to adjust the changes in motion represented in the image sequence.

[0031] The output unit 214 is configured to output various types of information. The output unit 214 is configured to perform output steps. For example, the output unit 214 outputs a second image sequence, which is an adjusted image sequence. The output unit 214 also outputs a fourth image sequence, which is a color-reduced image sequence.

[0032] The deletion unit 215 is configured to delete various types of information. The deletion unit 215 is configured to perform deletion steps. For example, when the second image sequence is determined, the deletion unit 215 deletes the data related to the second image sequence from the storage unit. Also, when the fourth image data is determined, the deletion unit 215 deletes the data related to the fourth image data from the storage unit.

[0033] The separation unit 216 is configured to separate various types of information. The separation unit 216 is configured to perform separation steps. For example, the separation unit 216 separates the line drawing portion and the filled portion from the third image data.

[0034] The color reduction unit 217 is configured to reduce the color of various types of information. The color reduction unit 217 is configured to perform a color reduction step. For example, the color reduction unit 217 clusters the colors contained in the third image data and reduces the color of the third image data according to the color-reduced information indicating the color corresponding to each cluster. Here, the color corresponding to each cluster is determined using the mode of each cluster. The color after color reduction is, for example, a color such as blue, yellow, or red, and in the following explanation, it will also be referred to as the type of color after color reduction.

[0035] The integration unit 218 is configured to integrate various types of information. The integration unit 218 is configured to perform integration steps. For example, the integration unit 218 integrates the line drawing portion with the color-reduced fill portion.

[0036] 3. Information Processing Methods Section 3 describes the flow of the information processing method of the information processing device 200 mentioned above. This information processing method comprises an acquisition step, a generation step, an adjustment step, an output step, a deletion step, a separation step, a color reduction step, and an integration step. Each of these steps is performed by the control unit 210 of the information processing device 200, as described in Section 2.

[0037] Figures 5 to 7 are activity diagrams showing the flow of the information processing method executed by the information processing device 200. The following will be explained in accordance with each activity in these activity diagrams. In this embodiment, the first image data and the second image data are assumed to be prepared in advance at the terminal 300. The first image data is assumed to be reference image data. The second image data is data showing the image after the first image data has been modified. Here, the first image data and the second image data are assumed to be colored image data. By using colored image data as input images in this manner, the accuracy of the final generated image sequence can be improved compared to conventional methods.

[0038] First, the control unit 310 in terminal 300 transmits the first image data and the second image data to the information processing device 200 (activity A110). The second image data is an example of the "state information" in the claim. The state information is at least one of the second image data showing the image after the change of the first image data, and a prompt showing the state after the change of the first image data. The prompt will be described later. In activity A110, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the first image data and the second image data from the storage unit 320. (2) The control unit 310 transmits the first image data and the second image data to the information processing device 200 via the communication unit 350.

[0039] Next, the control unit 210 in the information processing device 200 receives the first image data and the second image data from the terminal 300 (activity A120). In other words, in the acquisition step, the first image data and status information are acquired. In activity A120, for example, the following two stages of information processing are performed: (1) The communication unit 250 receives the first image data and the second image data from the terminal 300. (2) The control unit 210 stores the first image data and the second image data in the storage unit 220.

[0040] Next, the control unit 210 in the information processing device 200 generates interpolated image data (hereinafter also referred to as "interpolated image data") between the first image data and the second image data (Activity A130). In other words, in the generation step, multiple interpolated image data are generated, which are image data that interpolates between the first image data and the state information. In Activity A130, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the first image data and the second image data from the storage unit 220. (2) The control unit 210 executes the generation process and generates multiple interpolated image data. (3) The control unit 210 stores the multiple interpolated image data in the storage unit 220.

[0041] Next, the control unit 210 in the information processing device 200 generates an image sequence (hereinafter also referred to as the "first image sequence") arranged in chronological order by the first image data, multiple in-frame image data, and the second image data (Activity A140). In other words, the generation step generates the first image sequence, which is data arranged in chronological order by the first image data, multiple in-frame image data, and state information. In Activity A140, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the first image data, multiple in-frame image data, and the second image data from the storage unit 220. (2) The control unit 210 executes the generation process, arranging the first image data, multiple in-frame image data, and the second image data in chronological order to generate the first image sequence. (3) The control unit 210 stores the first image sequence in the storage unit 220.

[0042] Next, the control unit 210 in the information processing device 200 transmits the first image sequence to the terminal 300 (activity A150). In activity A150, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the first image sequence from the storage unit 220. (2) The control unit 210 transmits the first image sequence to the terminal 300 via the communication unit 250.

[0043] Next, the control unit 310 in terminal 300 receives the first image sequence from the information processing device 200 (activity A160). In activity A160, for example, the following two-stage information processing is performed: (1) The communication unit 350 receives the first image sequence from the information processing device 200. (2) The control unit 310 stores the first image sequence in the storage unit 320.

[0044] Next, the control unit 310 in terminal 300 displays the first image sequence on the display unit 330 (activity A170). In activity A170, for example, the following two-stage information processing is performed: (1) The control unit 310 reads the first image sequence from the storage unit 320. (2) The control unit 310 displays the first image sequence on the display unit 330.

[0045] Next, the control unit 310 in terminal 300 receives adjustment information, which is information for adjusting the changes in motion represented in the first image sequence (activity A180). Here, the adjustment information is information for adjusting the changes in motion represented in the first image sequence. The adjustment information also includes image count information, which is information indicating the number of image data contained in the second image sequence.

[0046] If the control unit 310 determines that it has received adjustment information, it proceeds to the processing of activity A190 (YES for activity A180). If the control unit 310 determines that it has not received adjustment information, it proceeds to the processing of activity A260 (NO for activity A180). In activity A180, for example, the following four stages of information processing are performed: (1) If the input unit 340 has received adjustment information, it transmits an acceptance signal to the control unit 310 via the communication bus 360. (2) If the input unit 340 has not received adjustment information, it transmits a non-acceptance signal to the control unit 310 via the communication bus 360. (3) If the control unit 310 has received an acceptance signal, it stores the adjustment information in the storage unit 320 and proceeds to the processing of activity A190. (4) If the control unit 310 has received a non-acceptance signal, it proceeds to the processing of activity A260.

[0047] Next, the control unit 310 in terminal 300 transmits adjustment information, which is information for adjusting the changes in motion represented in the first image sequence, to the information processing device 200 (Activity A190). In Activity A190, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the adjustment information from the storage unit 320. (2) The control unit 310 transmits the adjustment information to the information processing device 200 via the communication unit 350.

[0048] Figure 8 shows an example of a graph editor used when inputting adjustment information. The graph editor 410 displays a line graph 411, a setting area 412, and a graph area 413. The line graph 411 is a graph for adjusting the degree of change between the first image data A and the second image data B. The setting area 412 is an area for setting the number of image data (image count information) included in the second image sequence output after adjusting the change in motion. The graph area 413 is a variable area of ​​the line graph 411. The graph area 413 has the first image data A as its lower base and the second image data B as its upper base. That is, the closer the line graph 411 is to the lower base, the closer it is to the representation of the first image data A, and the closer it is to the upper base, the closer it is to the representation of the second image data B.

[0049] The shape of the line graph 411 can be changed by dragging and dropping the six movable points displayed. In the example shown in Figure 8, the graph gradually changes from the first image data A to the second image data B over time.

[0050] Thus, the adjustment information is information entered via the graph editor 410. This configuration allows for adjustment of changes in motion while visually confirming them. Furthermore, in this case, the graph editor 410 is an editor that uses a line graph to input adjustment information. This configuration allows for easier adjustment of changes in motion while visually confirming them.

[0051] Returning to the explanation of Figure 5, the control unit 210 in the information processing device 200 receives adjustment information from the terminal 300, which is information for adjusting the changes in motion represented in the first image sequence (Activity A200). In Activity A200, for example, the following two stages of information processing are performed: (1) The communication unit 250 receives the adjustment information from the terminal 300. (2) The control unit 210 stores the adjustment information in the storage unit 220.

[0052] Next, the control unit 210 in the information processing device 200 deletes some of the in-between image data included in the first image sequence based on the first image sequence and the input adjustment information (activity A210). In other words, in the adjustment step, the change in motion expressed in the first image sequence is adjusted based on the first image sequence and the input adjustment information. More specifically, in the adjustment step, the change in motion is adjusted by deleting some of the in-between image data from the first image sequence. With this configuration, the change in motion can be adjusted while reducing the data capacity of the image sequence. In addition, in the adjustment step, the number of image data included in the second image sequence is adjusted based on the number of image data information, which is information indicating the number of image data included in the second image sequence. With this configuration, the second image sequence can be changed quickly in a short time or slowly over a relatively long period of time, depending on the balance between the frame rate of the second image sequence and the number of image data included in the second image sequence.

[0053] Activity A210 performs, for example, the following three stages of information processing: (1) The control unit 210 reads the first image sequence and adjustment information from the storage unit 220. (2) The control unit 210 performs the adjustment process and deletes some of the in-between image data included in the first image sequence. (3) The control unit 210 stores the image sequence after the adjustment process (hereinafter also referred to as the "second image sequence") in the storage unit 220.

[0054] Figure 9 shows the process of generating a second image sequence after adjusting for motion changes from the first and second image data. First, the first image data A and the second image data B are prepared. Next, in-between image data 1 to 10 are generated between the first image data A and the second image data B. Then, in-between image data 2, 5, 8 and 9 are deleted to generate the second image sequence.

[0055] In the example in Figure 9, the frame rate of the first image sequence is set to 12 fps. The number of image data frames in the second image sequence is set to 8. The second image sequence can be shown to change in an accelerating manner from in-between image data 1 to 3, then from in-between image data 4 to 6, and then from in-between image data 7 to 10.

[0056] If the frame rate of the second image sequence is set to 12fps, the rate of change in the second image sequence will be accelerated by skipping parts where in-between image data has been deleted, based on the rate of change of the first image sequence. If the frame rate of the second image sequence is set to 8fps, the rate of change in the second image sequence will be slower overall, and the rate of change will be the same as that of the first image sequence by skipping parts where in-between image data has been deleted.

[0057] Returning to the explanation of Figure 5, the control unit 210 in the information processing device 200 then transmits the second image sequence to the terminal 300 (activity A220). In other words, in the output step, the second image sequence, which is the adjusted image sequence, is output. In activity A220, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the second image sequence from the storage unit 220. (2) The control unit 210 transmits the second image sequence to the terminal 300 via the communication unit 250.

[0058] Next, the control unit 310 in terminal 300 receives the second image sequence from the information processing device 200 (activity A230). In activity A230, for example, the following two-stage information processing is performed: (1) The communication unit 350 receives the second image sequence from the information processing device 200. (2) The control unit 310 stores the second image sequence in the storage unit 320.

[0059] Next, the control unit 310 in terminal 300 displays the second image sequence on the display unit 330 (activity A240). In activity A240, for example, the following two-stage information processing is performed: (1) The control unit 310 reads the second image sequence from the storage unit 320. (2) The control unit 310 displays the second image sequence on the display unit 330.

[0060] We will now move on to the explanation of Figure 6. Next, the control unit 310 in terminal 300 receives adjustment information, which is information for adjusting the changes in motion represented in the first image sequence (Activity A250). If the control unit 310 determines that it has received the adjustment information, it moves on to the processing of Activity A190 (Activity A250 YES). If the control unit 310 determines that it has not received the adjustment information, it moves on to the processing of Activity A260 (Activity A250 NO). In Activity A250, for example, the following four stages of information processing are executed: (1) If the input unit 340 has received the adjustment information, it transmits an acceptance signal to the control unit 310 via the communication bus 360. (2) If the input unit 340 has not received the adjustment information, it transmits a non-acceptance signal to the control unit 310 via the communication bus 360. (3) If the control unit 310 receives the acceptance signal, it stores the adjustment information in the storage unit 320 and moves on to the processing of Activity A190. (4) When the control unit 310 receives a non-received signal, it proceeds to the processing of activity A260.

[0061] Next, the control unit 310 in terminal 300 transmits setting information, including the number of colors after color reduction, to the information processing device 200 (activity A260). In this embodiment, the setting information is assumed to be the default setting. In activity A260, for example, the following two-stage information processing is performed: (1) The control unit 310 reads the setting information from the storage unit 320. (2) The control unit 310 transmits the setting information to the information processing device 200 via the communication unit 350.

[0062] Next, the control unit 210 in the information processing device 200 receives the configuration information from the terminal 300 (activity A270). In activity A270, for example, the following two-stage information processing is performed: (1) The communication unit 250 receives the configuration information from the terminal 300. (2) The control unit 210 stores the configuration information in the storage unit 220.

[0063] Next, the control unit 210 in the information processing device 200 separates each image data contained in the second image sequence into a line drawing portion and a filled portion (activity A280). In other words, in the separation step, the line drawing portion and the filled portion are separated from the image data. In activity A280, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the second image sequence and the image separation program from the storage unit 220. (2) The control unit 210 executes the separation process and separates each image data contained in the second image sequence into a line drawing portion and a filled portion. (3) The control unit 210 stores the line drawing portion and the filled portion (hereinafter also simply referred to as "line drawing portion" and "filled portion") of each image data in the storage unit 220.

[0064] Figure 10 is a diagram to supplement the processing of activity A280. Image data 510 is assumed to be any one of the image data included in the second image sequence. Activity A280 separates image data 510 into line drawing portion 520 and filled portion 530.

[0065] Returning to the explanation of Figure 6, the control unit 210 in the information processing device 200 then performs a color reduction process on the painted area (activity A290). In other words, in the color reduction step, the colors in the painted area (colors included in the third image data) are clustered, and the painted area (third image data) is color-reduced according to the color-reduced information indicating the color corresponding to each cluster. Here, the color corresponding to each cluster is determined using the mode of each cluster. The number of clusters in the clustering process corresponds to the number of color-reduced colors included in the setting information. That is, in the color reduction step, the painted area (image data) is color-reduced according to the set number of colors. With this configuration, the number of color-reduced colors can be arbitrarily set. Furthermore, in the color reduction step, first color-reduced information and second color-reduced information are generated according to the set number of colors. Here, both the first color-reduced information and the second color-reduced information indicate the color corresponding to each cluster and are equivalent in value.

[0066] In Activity A290, for example, the following three stages of information processing are performed: (1) The control unit 210 reads the setting information and the painted area from the storage unit 220. (2) The control unit 210 clusters the colors in the painted area and reduces the number of colors in the painted area according to the color corresponding to each cluster. (3) The control unit 210 stores the painted area after color reduction and the types of colors in the painted area after color reduction (hereinafter also referred to as "post-color reduction information") in the storage unit 220.

[0067] Next, the control unit 210 in the information processing device 200 transmits the color-reduced painted area and the type of color in the color-reduced painted area to the terminal 300 (Activity A300). In Activity A300, for example, the following three stages of information processing are performed: (1) The control unit 210 reads the color-reduced painted area and the second color-reduced information from the storage unit 220. (2) The control unit 210 transmits the color-reduced painted area and the second color-reduced information to the terminal 300 via the communication unit 250.

[0068] Next, the control unit 310 in terminal 300 receives the color-reduced painted area and the type of color in the color-reduced painted area from the information processing device 200 (Activity A310). In Activity A310, for example, the following two stages of information processing are performed: (1) The communication unit 350 receives the color-reduced painted area and the second color-reduced information from the information processing device 200. (2) The control unit 310 stores the color-reduced painted area and the second color-reduced information in the storage unit 320.

[0069] Next, the control unit 310 in terminal 300 displays the color-reduced painted area and the types of colors in the color-reduced painted area on the display unit 330 (activity A320). In other words, in the color reduction step, the color-reduced painted area and the second color-reduced information are displayed according to the set number of colors. In activity A320, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the color-reduced painted area and the second color-reduced information from the storage unit 320. (2) The control unit 310 displays the color-reduced painted area and the second color-reduced information on the display unit 330.

[0070] Next, the control unit 310 in terminal 300 accepts an operation to change the type of color after color reduction (second color reduction information) (Activity A330). If the control unit 310 determines that it has accepted the change operation, it proceeds to the processing of Activity A340 (Activity A330 YES). If the control unit 310 determines that it has not accepted the change operation, it proceeds to the processing of Activity A410 (Activity A330 NO). In Activity A330, for example, the following four stages of information processing are executed: (1) If the input unit 340 accepts the change operation, it transmits an acceptance signal to the control unit 310 via the communication bus 360. (2) If the input unit 340 does not accept the change operation, it transmits a non-acceptance signal to the control unit 310 via the communication bus 360. (3) If the control unit 310 receives the acceptance signal, it stores the change operation in the storage unit 320 and proceeds to the processing of Activity A340. (4) When the control unit 310 receives a non-received signal, it proceeds to the processing of activity A410.

[0071] Next, the control unit 310 in terminal 300 generates change information to change the type of color after color reduction (second post-color reduction information) (activity A340). In activity A340, for example, the following three stages of information processing are performed: (1) The control unit 310 reads the change operation from the storage unit 320. (2) The control unit 310 executes the generation process and generates change information to change the type of color after color reduction (second post-color reduction information). (3) The control unit 310 stores the change information in the storage unit 320.

[0072] Figures 11 to 14 are diagrams that supplement the processing of activities A310 to A340. Figure 11 shows the palette 600 displayed before the data received in activity A310 is read. The palette 600 displays the number of colors area 610, the color reduction method area 620, and the add button 630. In the number of colors area 610, the number of colors after color reduction can be set. In the color reduction method area 620, the algorithm for the color reduction process can be set, and here it is set to the algorithm of this embodiment (clustering process using the mode). The add button 630 is a button for increasing the number of colors after color reduction.

[0073] Figure 12 shows the palette 600 displayed after proceeding to the processing of activity A320. Palette 600 corresponds to the "second color reduction information" in the claim. Palette 600 displays the number of colors area 610, the color reduction method area 620, an add button 630, and a color type 640. Each color type 640 has a delete button 641. The delete button 641 is used to delete that color. Each color in the color type 640 is a color extracted by clustering using the mode.

[0074] Figure 13 shows the process moving to Activity A330, where the delete button 641 for three of the colors displayed in Color Type 640 is clicked, resulting in a reduced number of colors to 15. In the color reduction step, if any color is deleted from the reduced color range, the deleted color is further reduced from the painted area.

[0075] Figure 14(A) shows the state after the add button 630 was clicked in Figure 13, and color 642 was added. When color 642 is selected, the color setting box 650 is displayed as shown in Figure 14(B). The color setting box 650 displays areas 651, 652, and 653. Color 642 is set by selecting an arbitrary color in area 651, operating the slider in area 652, and entering a numerical value in area 653. In the color reduction step, if an arbitrary color is added to the color after reduction, the added color is added to the fill area.

[0076] Here, the first color-reduced information is stored in the storage unit 220 of the information processing device 200. Therefore, the first color-reduced information is not changed, and the colors corresponding to each cluster can be maintained.

[0077] Returning to the explanation of Figure 6, the control unit 310 in terminal 300 then transmits change information to the information processing device 200 to change the type of color after color reduction (second color reduction information) (Activity A350). In Activity A350, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the change information from the storage unit 320. (2) The control unit 310 transmits the change information to the information processing device 200 via the communication unit 350.

[0078] Next, the control unit 210 in the information processing device 200 receives change information from the terminal 300 to change the type of color after color reduction (second color reduction information) (Activity A360). In Activity A360, for example, the following two stages of information processing are performed: (1) The communication unit 250 receives the change information from the terminal 300. (2) The control unit 210 stores the change information in the storage unit 220.

[0079] We will now move on to the explanation of Figure 7. Next, the control unit 210 in the information processing device 200 reflects the type of color after color reduction included in the change information to the painted portion (activity A370). To put this in terms of steps, in the color reduction step, if the second color-reduced information is changed, the second color-reduced information after the change is reflected in the painted portion (image data). In activity A370, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the color-reduced painted portion and the change information from the storage unit 220. (2) The control unit 210 performs a reflection process and reflects the type of color after color reduction included in the change information to the painted portion. (3) The control unit 210 stores the painted portion after the reflection process (hereinafter also referred to as the "reflected painted portion") in the storage unit 220. With this configuration, it is possible to change the color corresponding to each cluster by changing the second color-reduced information while maintaining each cluster in the first color-reduced information.

[0080] Next, the control unit 210 in the information processing device 200 transmits the painted portion after the reflection process to the terminal 300 (activity A380). In activity A380, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the reflected painted portion from the storage unit 220. (2) The control unit 210 transmits the reflected painted portion to the terminal 300 via the communication unit 250.

[0081] Next, the control unit 310 in terminal 300 receives the painted portion after the reflection process from the information processing device 200 (activity A390). In activity A390, for example, the following two stages of information processing are performed: (1) The communication unit 350 receives the reflected painted portion from the information processing device 200. (2) The control unit 310 stores the reflected painted portion in the storage unit 320.

[0082] Next, the control unit 310 in terminal 300 accepts an operation to change the type of color after color reduction (second color reduction information) (Activity A400). If the control unit 310 determines that it has accepted the change operation, it proceeds to the processing of Activity A340 (Activity A400 YES). If the control unit 310 determines that it has not accepted the change operation, it proceeds to the processing of Activity A410 (Activity A400 NO). In Activity A400, for example, the following four stages of information processing are executed: (1) If the input unit 340 accepts the change operation, it transmits an acceptance signal to the control unit 310 via the communication bus 360. (2) If the input unit 340 does not accept the change operation, it transmits a non-acceptance signal to the control unit 310 via the communication bus 360. (3) If the control unit 310 receives the acceptance signal, it stores the change operation in the storage unit 320 and proceeds to the processing of Activity A340. (4) When the control unit 310 receives a non-received signal, it proceeds to the processing of activity A410.

[0083] Next, the control unit 310 in terminal 300 transmits an termination signal, which is a signal to terminate the color reduction process, to the information processing device 200 (activity A410). The termination signal is generated, for example, when a termination operation is performed on palette 600. In activity A410, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the termination signal from the storage unit 320. (2) The control unit 310 transmits the termination signal to the information processing device 200 via the communication unit 350.

[0084] Next, the control unit 210 in the information processing device 200 receives an termination signal from the terminal 300, which is a signal to terminate the color reduction process (activity A420). In activity A420, for example, the following two stages of information processing are performed: (1) The communication unit 250 receives the termination signal from the terminal 300. (2) The control unit 210 stores the termination signal in the storage unit 220.

[0085] Next, the control unit 210 in the information processing device 200 integrates the line drawing portion of each image data included in the second image sequence with the colored portion of each image data after the reflection processing (Activity A430). In other words, the integration step integrates the line drawing portion with the colored portion after color reduction. In Activity A430, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the line drawing portion and the reflected colored portion from the storage unit 220. (2) The control unit 210 executes the integration step and integrates the line drawing portion and the reflected colored portion. (3) The control unit 210 stores the integrated image data (hereinafter also referred to as "fourth image data") in the storage unit 220.

[0086] Figure 15 is a diagram illustrating the processing of Activity A430. Activity A430 integrates the line drawing portion 520 and the color-reduced filled portion 540 to generate integrated image data 550 (the fourth image data). The line drawing portion 520, the color-reduced filled portion 540, and the integrated image data 550 are assumed to be any one of each image data. Activity A430 integrates the line drawing portion 520 and the color-reduced filled portion 540 of each image data in the second image sequence, and generates an integrated image sequence with each integrated image data 550 as the image sequence.

[0087] Next, the control unit 210 in the information processing device 200 transmits the integrated image sequence to the terminal 300 (activity A440). In other words, in the output step, the fourth image data, which is the color-reduced image data, is output. In activity A440, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the integrated image sequence from the storage unit 220. (2) The control unit 210 transmits the integrated image sequence to the terminal 300 via the communication unit 250.

[0088] Next, the control unit 210 in the information processing device 200 deletes the data relating to the second image sequence and the data relating to the fourth image data from the storage unit 220 (activity A450). In other words, in the deletion step, when the second image sequence is determined, the data relating to the second image sequence is deleted from the storage unit. This configuration protects information relating to the image sequence in which the change in motion has been determined. Also, in the deletion step, when the fourth image data is determined, the data relating to the fourth image data is deleted from the storage unit. This configuration protects information relating to the image data after color reduction processing.

[0089] In Activity A450, for example, the following two-stage information processing is performed: (1) The control unit 210 reads data relating to the second image sequence and data relating to the fourth image data from the storage unit 220. (2) The control unit 210 performs a deletion process and deletes the data relating to the second image sequence and data relating to the fourth image data from the storage unit 220.

[0090] Next, the control unit 310 in terminal 300 receives the integrated image sequence from the information processing device 200 (activity A460). In activity A460, for example, the following two stages of information processing are performed: (1) The communication unit 350 receives the integrated image sequence from the information processing device 200. (2) The control unit 310 stores the integrated image sequence in the storage unit 320.

[0091] Next, the control unit 310 in terminal 300 displays the integrated image sequence on the display unit 330 (activity A470). In activity A470, for example, the following two-stage information processing is performed: (1) The control unit 310 reads the integrated image sequence from the storage unit 320. (2) The control unit 310 displays the integrated image sequence on the display unit 330.

[0092] Figures 16 and 17 show a comparison between image data processed with color reduction using the method of this embodiment and image data processed with color reduction using a conventional method. Original image 700 is image data before color reduction. Comparative example data 710 is image data processed with color reduction using a conventional method (k-means method). Comparative example data 720 is image data processed with color reduction using another conventional method (DBSCAN). Example data 730 is image data processed with color reduction using the method of this embodiment.

[0093] Now, looking at Figure 17, which is an enlarged view of region 740 in Figure 16, we see that color noise was generated in comparative example data 710 and comparative example data 720. On the other hand, no color noise was generated in example data 730, similar to the original image 700. This shows that the color reduction process of this embodiment can improve the color reproducibility after the color reduction process compared to conventional methods.

[0094] According to the information processing method of this embodiment, it is possible to adjust the changes in motion represented in the generated image sequence. Furthermore, according to the information processing method of this embodiment, the color reproduction after color reduction can be improved compared to conventional methods. Moreover, due to its simple configuration, the resources saved can be used for other core functions.

[0095] Although embodiments of the present invention have been described above, the present invention is not limited thereto and can be modified as appropriate without departing from the technical spirit of the invention.

[0096] 4. Variations Section 4 describes modifications of this embodiment. The following modifications can be combined as appropriate.

[0097] An embodiment of this design may be a program. This program is configured to cause a computer to execute each step of the information processing method of this design.

[0098] The control unit 210 writes (stores) and reads various data and information to the storage unit 220, but is not limited to this. For example, it may also use registers or cache memory within the control unit 210 to perform information processing for each activity.

[0099] In this embodiment, the second image data is described as being a single image, but it is not limited to this. The second image data may consist of two or more images. In this case, in-between image data is generated between the first image data and each of the second image data.

[0100] Figure 18 illustrates prompts, which are an example of state information. Figure 18(A) shows the input box for a prompt. Figure 18(B) shows an example of a graph editor corresponding to a prompt.

[0101] If status information is used as the prompt, an arbitrary prompt is entered into the input box 420 in Activity A110. Here, we assume that "Angry" is entered. Subsequently, the graph editor 430 is displayed in Activity A180. The graph editor 430 displays a line graph 431, a settings area 432, and a graph area 433.

[0102] The line graph 431 is a graph for adjusting the degree of change between the first image data A and Angry. The setting area 432 is an area for setting the number of image data (image count information) included in the second image sequence output after adjusting the change in motion. The graph area 433 is the variable area of ​​the line graph 431. The graph area 433 has the first image data A as its lower base and Angry as its upper base. That is, the closer the line graph 431 is to the lower base, the closer it is to the first image data A, and the closer it is to the upper base, the closer it is to the Angry.

[0103] In the example in Figure 18, assuming the first image data A represents a normal facial expression, the image gradually changes to an angry expression over time. Here, the type of prompt is not particularly limited as long as it is in natural language. For example, it could be "sorrow" to indicate sadness, "joy" to indicate happiness, or any other natural language besides English.

[0104] Figure 19 shows an example of setting a line graph in the graph editor. The graph editor 440 displays a line graph 441, a setting area 442, and a graph area 443. The line graph 441 shows a state in which the image changes rapidly from the first image data A to the vicinity of the second image data B, then rapidly from the vicinity of the second image data B to the vicinity of the first image data A, then gradually from the vicinity of the first image data A to the second image data B side, and then from the second image data B side to the first image data A side. In this way, the adjustment information may be information for adjusting the movement of changing from the first image data to the state information, and the movement of changing from the state information to the first image data. With this embodiment, it is possible to adjust changes in movements that require repetition, such as blinking or lip-syncing.

[0105] The type of color after modification included in the modification information generated in Activity A350 may also be applicable to image data other than the third or fourth image data. In this case, the type of color after modification may be stored in the storage unit 220 of the information processing device 200, in the storage unit 320 of the terminal 300, or on an external hard disk drive. According to this embodiment, the type of color set as the color after color reduction can be applied to other images as well.

[0106] In this embodiment, the color-reduced information is described as first color-reduced information and second color-reduced information, but it is not limited to this. In the color reduction step, the color-reduced information may be displayed according to the set number of colors. In the color reduction step, if the color-reduced information is changed, the changed color-reduced information may be reflected in the painted portion (image data). With this embodiment, the color after color reduction can be changed arbitrarily.

[0107] In this embodiment, adjustment information is shown as being input by manipulating a line graph, but it is not limited to this. For example, it may be input by manipulating a bar graph or a pie chart, or by inputting something other than a graph, such as a predetermined index.

[0108] Activities A110 to A250 and activities A260 to A470 may be executed independently. In this case, in activity A260, the colored image data may be sent to the information processing device 200, and in activity A270, the colored image data may be received. To put this in terms of steps, in the acquisition step, a third image data, which is the colored image data, is acquired.

[0109] In this embodiment, the information processing method of this embodiment has been described as being executed between the information processing device 200 and the terminal 300, but it is not limited to this. For example, if the program of this embodiment is installed on the terminal 300, the information processing method of this embodiment may be executed on the terminal 300 alone.

[0110] In this embodiment, a CPU is given as an example of the control unit 210 in the information processing device 200 and the control unit 310 in the terminal 300, but it is not limited to this. The control unit 210 and the control unit 310 may be a CPU, a Graphics Processing Unit (GPU), a Neural Processing Unit (NPU), or a Tensor Processing Unit (TPU), respectively, or a combination of these processors. In other words, the control unit 210 and the control unit 310 refer to one or more of the above processors, and the information processing method of this embodiment may be executed by having these processors cooperate.

[0111] 5. Others The product may be provided in any of the following embodiments.

[0112] (1) An information processing method comprising an acquisition step, a generation step, an adjustment step, and an output step, wherein the acquisition step acquires first image data and state information, the state information being at least one of a second image data showing an image after a change in the first image data and a prompt showing the state after a change in the first image data; the generation step generates a plurality of in-between image data which are image data that interpolates between the first image data and the state information; the generation step generates a first image sequence which is data arranged in chronological order by the first image data, the plurality of in-between image data, and the state information; the adjustment step adjusts the change in motion expressed in the first image sequence based on the first image sequence and input adjustment information, the adjustment information is information for adjusting the change in motion; and the output step outputs a second image sequence which is the adjusted image sequence.

[0113] This configuration allows for adjustment of the changes in motion represented in the generated image sequence. Furthermore, its simple configuration allows the saved resources to be used for other core functions.

[0114] (2) The information processing method described in (1) above, wherein the adjustment information is information input via a graph editor.

[0115] In this configuration, changes in movement can be adjusted while being visually confirmed.

[0116] (3) The information processing method described in (2) above, wherein the graph editor is an editor that inputs the adjustment information using a line graph.

[0117] This configuration allows for easier adjustment of changes in movement while visually confirming them.

[0118] (4) An information processing method according to any one of (1) to (3) above, wherein in the adjustment step, the change in motion is adjusted by deleting some of the intermediate image data from the plurality of intermediate image data from the first image sequence.

[0119] According to this configuration, the data size of the image sequence can be reduced and changes in motion can be adjusted.

[0120] (5) An information processing method according to any one of (1) to (4) above, wherein the adjustment information is information for adjusting the movement that changes from the first image data toward the state information, and the movement that changes from the state information toward the first image data.

[0121] This configuration allows for adjustment of changes in movements that require repetition, such as blinking or lip-syncing.

[0122] (6) An information processing method according to any one of (1) to (5) above, wherein the adjustment information includes image count information which is information indicating the number of image data included in the second image sequence, and the adjustment step adjusts the number of image data included in the second image sequence based on the image count information.

[0123] In this configuration, by balancing the frame rate of the second image sequence with the number of image data points included in the second image sequence, the second image sequence can be changed rapidly in a short time or slowly over a relatively long period of time.

[0124] (7) An information processing method according to any one of (1) to (6) above, further comprising a deletion step, wherein in the deletion step, when the second image sequence is determined, data relating to the second image sequence is deleted from the storage unit.

[0125] According to this embodiment, information regarding the sequence of images that confirms the change in motion can be protected.

[0126] (8) An information processing method according to any one of (1) to (7) above, wherein the image data is colored image data.

[0127] This configuration makes it possible to improve the accuracy of the final generated image sequence compared to conventional methods.

[0128] (9) An information processing system comprising a control unit, wherein the control unit is configured to perform each step of the information processing method described in any one of (1) to (8) above.

[0129] This configuration allows for adjustment of the changes in motion represented in the generated image sequence. Furthermore, its simple configuration allows the saved resources to be used for other core functions.

[0130] (10) A program configured to cause a computer to perform each step of the information processing method described in any one of (1) to (8) above.

[0131] This configuration allows for adjustment of the changes in motion represented in the generated image sequence. Furthermore, its simple configuration allows the saved resources to be used for other core functions. Of course, this is not always the case. [Explanation of Symbols]

[0132] 1: In-between image data 2: In-between image data 3: In-between image data 4: In-between image data 5: In-between image data 6: In-between image data 7: In-between image data 8: In-between image data 9: In-between image data 10: In-between image data 100: Information Processing Systems 200: Information Processing Device 210: Control Unit 211: Acquisition Department 212 :Generation part 213: Adjustment section 214: Output section 215: Deleted section 216: Separation section 217: Color reduction part 218: Integration Department 220: Storage section 250: Communications Department 260: Communications bus 300: Terminal 310: Control Unit 320: Storage section 330: Display section 340: Input section 350: Communications Department 360: Communications Bus 410: Graph Editor 411: Line graph 412: Settings area 413: Graph area 420: Input box 430: Graph Editor 431: Line graph 432: Settings area 433: Graph area 440: Graph Editor 441: Line graph 442: Settings area 443: Graph area 510: Image data 520: Line drawing portion 530: Painted area 540: Painted area 550: Integrated image data 600: Palette 610: Color range 620: Color reduction method area 630: Add button 640: Type 641: Delete button 642: Color 650: Color setting box 651 :Area 652: area 653: area 700: Original image 710: Comparative Example Data 720: Comparative Example Data 730: Example Data 740: area A: First image data B: Second image data

Claims

1. Information processing method, It comprises an acquisition step, a generation step, an adjustment step, and an output step, In the acquisition step described above, first image data and status information are acquired. The state information is at least one of a second image data showing the image after the first image data has been changed, and a prompt showing the state after the first image data has been changed. In the generation step, a plurality of intermediate image data are generated, which are image data that interpolates between the first image data and the state information. In the generation step, a first image sequence is generated, which is data in which the first image data, the plurality of in-between image data, and the state information are arranged in chronological order. In the adjustment step, the changes in motion represented in the first image sequence are adjusted based on the first image sequence and the input adjustment information. The aforementioned adjustment information is information for adjusting the change in the aforementioned movement, In the output step, a second image sequence, which is the adjusted image sequence, is output. Information processing methods.

2. In the information processing method described in claim 1, The aforementioned adjustment information is information entered via the graph editor. Information processing methods.

3. In the information processing method described in claim 2, The aforementioned graph editor is an editor that inputs the adjustment information using a line graph. Information processing methods.

4. In the information processing method described in claim 1, In the adjustment step, the change in motion is adjusted by deleting some of the intermediate image data from the plurality of intermediate image data from the first image sequence. Information processing methods.

5. In the information processing method described in claim 1, The adjustment information is information for adjusting the movement that changes from the first image data to the state information, and the movement that changes from the state information to the first image data. Information processing methods.

6. In the information processing method described in claim 1, The adjustment information includes image count information, which is information indicating the number of image data contained in the second image sequence. In the adjustment step, the number of image data included in the second image sequence is adjusted based on the number of images information. Information processing methods.

7. In the information processing method described in claim 1, Furthermore, it includes a deletion step, In the deletion step, if the second image sequence is determined, the data relating to the second image sequence is deleted from the storage unit. Information processing methods.

8. In the information processing method described in claim 1, The aforementioned image data is colored image data. Information processing methods.

9. An information processing system, Equipped with a control unit, The control unit is configured to perform each step of the information processing method described in any one of claims 1 to 8. Information processing system.

10. It is a program, The information processing method described in any one of claims 1 to 8 is configured to cause a computer to perform each step of the information processing method, program.

Images