Method, program, information processing unit and system for supporting evaluator to subjectively evaluate plural evaluation objects

Abstract

To provide a method, a program, an information processing unit and a system for supporting an objectively correct evaluation of plural evaluation objects.SOLUTION: A method done by an information processing unit 1 includes: a step for displaying a marker arrangement area 6, on which plural markers (M1 to M7) corresponding to data one by one are arranged, onto a display device; a step for moving one marker selected from the plural markers (M1 to M7) according to a direction input by an evaluator; a step for replaying one evaluation object data corresponding to the one marker selected from the plural markers (M1 to M7) on the display device according to a direction input by an evaluator; and a step for acquiring positions of the markers in the marker arrangement area 6 as a result of evaluations of plural evaluation objects.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method, a program, an information processing device, and a system for supporting an evaluator in giving an evaluation to an evaluation target. [Background technology]

[0002] There are known devices for evaluating sounds generated by industrial products. For example, the following patent document describes a sound quality evaluation device that calculates a sound loudness index, an index related to broadband noise, an index related to the high frequency weighting of pure tone components, and the like, and has a means for calculating a comprehensive sound quality index from these multiple acoustic indices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2001-336975 Summary of the Invention [Problem to be solved by the invention]

[0004] The evaluation results obtained by the above-mentioned sound quality evaluation devices may not match the human auditory sensation, so a subjective sensory evaluation by a human evaluator is usually also required.

[0005] However, in sensory evaluation by humans, it is difficult to give absolute evaluation results like an evaluation device that calculates evaluation values ​​from physical quantities. Therefore, when performing sensory evaluation on multiple evaluation objects collectively, it is easier to obtain subjectively correct evaluation results by giving a relative evaluation based on a comparison between the evaluation objects, rather than giving an evaluation for each evaluation object individually. However, when performing sensory evaluation on multiple sound sources using the semantic differential method (SD method), it is common to play the multiple sound sources one by one in order and evaluate each sound source in order. In other words, it is difficult to give a relative evaluation based on a comparison between the evaluation objects in a sequential method in which evaluation is performed one by one in order.

[0006] For example, if one evaluation target is given the highest rating, and then another evaluation target is found to be rated higher than the first evaluation target, in which case the method of rating each target in order would result in the second evaluation target receiving the same rating (the highest rating) as the first evaluation target, and would not be able to obtain an evaluation result that correctly reflects the difference in subjective impressions between the evaluation targets.

[0007] The present invention has been made in consideration of the above circumstances, and its general purpose is to provide a method, program, information processing device, and system that support the giving of more subjectively correct evaluations of multiple evaluation targets. [Means for solving the problem]

[0008] A first aspect of the present invention is a method in which an information processing device assists an evaluator in providing a subjective evaluation to each of a plurality of evaluation objects, the evaluation objects being perceptual objects including at least one of sound, still image, and video obtained by playing evaluation object data including at least one of sound data, still image data, and video data on a playback device, the method including the steps of: displaying, on a display device, a marker placement area in which a plurality of markers corresponding one-to-one to the plurality of evaluation object data are placed by the information processing device; moving one marker selected from the plurality of markers in response to an instruction input by the evaluator; and displaying, on a display device, a marker placement area in which a plurality of markers corresponding to the plurality of evaluation object data are placed by the information processing device in response to an instruction input by the evaluator. a step of playing back data for one evaluation target on a playback device; and a step of an information processing device acquiring evaluation results for the multiple evaluation targets, wherein the step of displaying a marker placement area includes displaying a marker placement area including one or more indicator guide elements that guide, for each of one or more indicators related to the evaluation of the evaluation target, a specified relationship between a marker position in the marker placement area and an index value, and the step of acquiring an evaluation result for one evaluation target includes acquiring the positions of each of the multiple markers in the marker placement area, or acquiring the positions of each of the multiple markers in the marker placement area and acquiring one or more index values ​​for each of the multiple evaluation targets based on the acquired position of each marker and the specified relationship.

[0009] A second aspect of the present invention is a program including instructions for causing an information processing device to perform processing to assist an evaluator in providing subjective evaluations for each of a plurality of evaluation objects, wherein the evaluation objects are perceptual objects including at least one of sound, still images, and video obtained by playing evaluation object data including at least one of sound data, still image data, and video data on a playback device, and the processing performed by the information processing device in accordance with the instructions includes each step of the method relating to the first aspect above.

[0010] A third aspect of the present invention is an information processing device that supports an evaluator in providing subjective evaluations for each of a plurality of evaluation objects, wherein the evaluation objects are perceptual objects including at least one of sound, still images, and video obtained by playing evaluation object data including at least one of sound data, still image data, and video data on a playback device, and the information processing device has a processing unit and a memory unit that stores instructions to be executed by the processing unit, and the processing performed by the processing unit in accordance with the instructions includes each step of the method relating to the first aspect above.

[0011] A fourth aspect of the present invention is an information processing device that supports an evaluator in providing subjective evaluations for each of a plurality of evaluation objects, wherein the evaluation objects are perceptual objects including at least one of sound, still images, and video obtained by playing evaluation object data including at least one of sound data, still image data, and video data on a playback device, and the information processing device has means for performing each step of the method relating to the first aspect above.

[0012] A fifth aspect of the present invention is a system for assisting an evaluator in providing subjective evaluations for each of a plurality of evaluation objects, the evaluation objects being perceptual objects including at least one of sound, still images, and video obtained by playing evaluation object data including at least one of sound data, still image data, and video data on a playback device, the system comprising one or more information processing devices and a server device, the information processing devices having means for performing each step of the method according to the first aspect above, and the server device collecting evaluation results obtained in a step of obtaining evaluation results for the plurality of evaluation objects from the one or more information processing devices. Effect of the Invention

[0013] According to the present invention, it is possible to provide a method, a program, an information processing device, and a system that support giving more subjectively correct evaluations to a plurality of evaluation targets. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a diagram showing an example of the configuration of a system according to the present embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an evaluation input screen displayed on the information processing device. [Diagram 3] FIG. 3 is a diagram showing an example of the evaluation input screen. [Figure 4] FIG. 4 is a diagram showing an example of the evaluation input screen. [Diagram 5] FIG. 5 is a diagram showing an example of the evaluation input screen. [Figure 6] FIG. 6 is a diagram showing an example of the evaluation input screen. [Figure 7] FIG. 7 is a diagram showing an example of the evaluation input screen. [Figure 8] FIG. 8 is a diagram showing an example of the evaluation input screen. [Figure 9] FIG. 9 is a diagram showing an example of a screen for setting a playback target section. [Figure 10] FIG. 10 is a flowchart illustrating an example of a process for inputting various settings. [Figure 11] FIG. 11 is a first flowchart illustrating an example of a process for inputting an evaluation. [Figure 12] FIG. 12 is a second flowchart illustrating an example of the process of inputting an evaluation. [Figure 13] FIG. 13 is a third flowchart illustrating an example of the process of inputting an evaluation. [Figure 14] FIG. 14 is a fourth flowchart illustrating an example of a process for inputting an evaluation. [Figure 15] FIG. 15 is a diagram showing an example of a marker arrangement area when there is one evaluation item. [Figure 16] FIG. 16 is a diagram showing an example of a marker arrangement area when there are multiple evaluation items. [Figure 17] FIG. 17 is a diagram showing an example of a marker arrangement area when there are multiple evaluation items. [Figure 18] 18A to 18H are diagrams showing setting examples of the index guide element. [Figure 19] 19A to 19C are diagrams showing examples of graphs showing the evaluation results. [Figure 20] FIG. 20 is a diagram showing an example of a graph showing the evaluation results. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Fig. 1 is a diagram showing an example of a system according to the present embodiment. The system according to the present embodiment performs processing to support an evaluator in inputting a subjective evaluation of an evaluation target such as sound. The system shown in Fig. 1 has a server device 2 and one or more information processing devices 1 that can communicate with each other via a communication network 9 such as the Internet. The information processing device 1 is an example of an information processing device of the present invention.

[0016] The evaluation object in this embodiment is an object of human perception, and is, for example, an object of perception including at least one of sound, still image, and video. The evaluation object is perceived by an evaluator by playing evaluation object data (sound data, still image data, video data, etc.) on a playback device (speaker, display, etc.).

[0017] [Information processing device 1] For example, the information processing device 1 is a device equipped with an information communication function such as a personal computer, tablet, or smartphone, and is operated by an evaluator who evaluates an evaluation target such as sound. In the example of FIG. 1, the information processing device 1 has a communication unit 11, an input unit 12, a display unit 13, a sound output unit 14, a storage unit 15, and a processing unit 16. The sound output unit 14 is an example of a playback device of the present invention. The storage unit 15 is an example of a storage unit of the present invention. The processing unit 16 is an example of a processing unit of the present invention.

[0018] The communication unit 11 communicates with other devices (such as the server device 2) via the communication network 9. The communication unit 11 includes a communication device (such as a network interface card) that communicates in accordance with a predetermined communication standard (such as wireless LAN or Ethernet (registered trademark)).

[0019] The input unit 12 inputs instructions according to user operations and other information to the processing unit 16. For example, the input unit 12 includes at least one of devices having an input function, such as a button, a keyboard, a mouse, a touchpad, a touch panel, a microphone, and a camera.

[0020] The display unit 13 is a device that displays an image corresponding to the video signal generated in the processing unit 16, and includes display devices such as a liquid crystal display, an organic EL display, and a projector.

[0021] The sound output unit 14 is a device that outputs sound corresponding to the acoustic signal generated in the processing unit 16, and includes, for example, a speaker.

[0022] The storage unit 15 stores one or more programs 151 including instructions executable by the processing unit 16, data temporarily stored during processing by the processing unit 16, data used in processing by the processing unit 16, data obtained as a result of processing by the processing unit 16, and the like. The storage unit 15 may include, for example, a main storage device (ROM, RAM, etc.) and an auxiliary storage device (flash memory, SSD, hard disk, optical disk, etc.). The storage unit 15 may be composed of one storage device or may be composed of multiple storage devices. When the storage unit 15 is composed of multiple storage devices, each storage device is connected to the processing unit 16 via a computer bus or any other data communication means.

[0023] The processing unit 16 generally controls the overall operation of the information processing device 1 and executes a predetermined information processing. The processing unit 16 includes, for example, one or more processors (such as a central processing unit (CPU), a micro-processing unit (MPU), or a digital signal processor (DSP)) that execute processing in accordance with instructions of one or more programs 151 stored in the storage unit 15. The processing unit 16 operates as a computer by the one or more processors executing instructions of one or more programs 151 stored in the storage unit 15. The information processing device 1 may include a plurality of such computers, and these computers may execute processing in cooperation with each other by communicating with each other via an arbitrary communication network.

[0024] The processing unit 16 may include one or more dedicated hardware (such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)) configured to realize a specific function. In this case, the processing unit 16 may execute all of the processes described in this embodiment in a computer, or may execute at least a part of the processes in dedicated hardware.

[0025] The program 151 may be recorded, for example, in a computer-readable recording medium (such as an optical disk, a memory card, a USB memory, or other non-transitory tangible medium). The processing unit 16 may read at least a part of one or more programs 151 recorded in such a recording medium using a recording medium reading device (such as an optical disk device) or an interface device (such as a USB interface) (not shown), and write the part in the storage unit 15.

[0026] Alternatively, the processing unit 16 may receive at least a part of one or more programs 151 from another device connected to the communication network 9 via the communication unit 11, and write the program 15 to the storage unit 15. The program 151 may include an instruction to cause the processing unit 16 to perform at least a part of the processing according to the present embodiment, which will be described later. For example, if the server device 2 is a server that provides a web application, and an evaluation input screen, which will be described later, is a screen on a web browser provided by the server device 2, at least a part of the program 151 may include code for the web browser provided by the server device 2 (code such as HTML, CSS, JavaScript (registered trademark), etc.).

[0027] [Server device 2] The server device 2 is a device (such as a cloud server) that performs processing to collect and analyze the evaluation results of the evaluators on the evaluation object (such as sound) from the information processing device 1, and is equipped with, for example, a communication unit, a memory unit, and a processing unit similar to the communication unit 11, memory unit 15, and processing unit 16 of the information processing device 1 described above.

[0028] The server device 2 may also be a web application server that communicates with the information processing device 1 on which a web browser runs and executes a predetermined process via the web browser. That is, the server device 2 may provide the information processing device 1 with a code for the web browser (HTML, CSS, JavaScript (registered trademark), etc.) so that the information processing device 1 performs processes related to the input of evaluations (displaying a screen for inputting evaluations, playing the evaluation target (sound, etc.), inputting evaluation results, etc.) on the web browser.

[0029] Here, the process of inputting an evaluation for an evaluation target in the system shown in Fig. 1 having the above-mentioned configuration (hereinafter, may be referred to as "evaluation input process") will be described with reference to the drawings. In the following example, the process when the evaluation target is sound will be described.

[0030] Fig. 2 is a diagram showing an example of an evaluation input screen 5 displayed on the display unit 13 of the information processing device 1. The evaluation input screen 5 shown in Fig. 2 has a marker placement area 6, a marker display setting area 7, a seek bar 51, a playback mode button 52, and buttons 53 and 54.

[0031] A number of markers (M1 to M7) that correspond one-to-one to a number of evaluation target data are placed in the marker placement area 6. The marker placement area 6 shown in the example of Fig. 2 includes a rectangular area 60 surrounded by rectangular lines, and the markers M1 to M7 are placed in the rectangular area 60. Hereinafter, any one of the markers M1 to M7 will be referred to as "marker M". Each marker M has a number (1 to 7) written therein that indicates the evaluation target data that corresponds to the marker M.

[0032] The position of the marker M in the rectangular region 60 is associated with the values ​​of two indices related to the evaluation of the evaluation target. In the example of Fig. 2, the value of the "like / dislike index" represented by the two antonyms "like / dislike" and the value of the "strength index" represented by the two antonyms "strong / weak" are each associated with the position of the marker M.

[0033] The marker placement area 6 includes an index guide element that guides the relationship between the position of the marker M in the rectangular area 60 described above and the value of the index for each of the two indices (likes / dislikes index and strength index). 2, the word "like" added to the upper end of frame line 61 extending vertically of rectangular area 60 and the word "dislike" added to the lower end of frame line 61 are index guide elements that guide the user to a predetermined relationship (hereinafter sometimes referred to as "position-index relationship") between the value of the like / dislike index and the vertical position of marker M. These index guide elements related to the like / dislike index guide the user that the higher the position of marker M is, the greater the degree of "like," and the lower the position of marker M is, the greater the degree of "dislike." 2, the word "strong" added to the right end of the frame line 62 extending horizontally of the rectangular region 60, and the word "weak" added to the left end of the frame line 62 are indicator guide elements that guide the evaluator to a predetermined relationship (position-indicator relationship) between the value of the strength index and the horizontal position of the marker M. These indicator guide elements related to the strength index guide the evaluator that the further to the right the position of the marker is, the greater the degree of "strong," and the further to the left the position of the marker is, the greater the degree of "weak."

[0034] The information processing device 1 selects a specific marker M in response to an instruction input by the evaluator (for example, an instruction by operating the pointer 50), and moves the selected marker M in the marker placement area 6. For example, the information processing device 1 moves the marker M in response to a drag operation by the pointer 50. The information processing device 1 also selects a specific marker M in response to an instruction input by the evaluator (for example, an instruction by operating the pointer 50), and plays the sound of the evaluation target data corresponding to the selected marker M in the sound output unit 14. For example, the information processing device 1 selects the marker M in response to an operation of clicking the marker M with a mouse, an operation of tapping the marker M on a touch panel, an operation of placing the pointer on the marker M, or the like, and plays the sound of the evaluation target data corresponding to the selected marker M (hereinafter, sometimes abbreviated as "sound of marker M") in the sound output unit 14. While playing the sound of the marker M, the information processing device 1 displays the playback status (the ratio of the played time to the remaining playback time) by the seek bar 51.

[0035] For example, the information processing device 1 may simultaneously move the marker M and play the sound of the marker M. This allows the user to determine the value of the index by moving the position of the marker M while listening to the sound of the marker M (while perceiving the played evaluation target), making it easier to perform accurate evaluation.

[0036] When the information processing device 1 is reproducing evaluation target data corresponding to one marker M selected from a plurality of markers M, and an instruction from the evaluator to cancel the selection of the one marker is input, the information processing device 1 stops the reproduction of the evaluation target data. For example, when the information processing device 1 is reproducing evaluation target data of one marker M, and an operation to select a location in the marker placement area 6 where no marker M exists (such as an operation of clicking with a mouse) is performed, the information processing device 1 cancels the selection of the one marker M and stops the reproduction of the evaluation target data. Furthermore, when the information processing device 1 is reproducing evaluation target data of one marker M, and another marker M is selected by an instruction from the evaluator, the information processing device 1 may cancel the selection of the one marker M and stop the reproduction of the evaluation target data, and may also reproduce the evaluation target data of the other marker M.

[0037] When the evaluator presses the button 53, the information processing device 1 ends the evaluation of the marker M on the evaluation input screen 5 and acquires the evaluation results of the evaluation objects corresponding to each marker M. That is, the information processing device 1 acquires the positions of the multiple markers M in the marker placement area 6 as the evaluation results. The information processing device 1 transmits the acquired evaluation results (the positions of each marker M) to the server device 2. The server device 2 acquires the values ​​of two indices (like / dislike index, strength index) for each evaluation object based on the positions of each marker M received from the information processing device 1 and the "position-index relationship" of the two indices.

[0038] Alternatively, the information processing device 1 may acquire the values ​​of two indices (like / dislike index and strength index) for each evaluation object based on the acquired position of each marker M and the "position-index relationship" of the two indices. In this case, the information processing device 1 may transmit the two indices acquired for each evaluation object to the server device 2 as the evaluation result of each evaluation object.

[0039] When the evaluator presses the button 54, the information processing device 1 stops inputting the evaluation on the evaluation input screen 5 and ends the evaluation input process.

[0040] In this way, on the evaluation input screen 5, the sound of the marker M placed in the marker placement area 6 is played in response to the evaluator's instruction. In addition, the "position-index relationship" in the marker placement area 6 is guided to the evaluator by the index guide element, and the marker M is moved in response to the instruction of the evaluator who recognizes this guidance. Then, as the evaluation result for each evaluation target, the position of each marker M in the marker placement area 6, or the value of the index of each evaluation target based on the position of each marker M and the "position-index relationship" is acquired. This allows the evaluator to evaluate the sound of each marker M while comparing the index values ​​of each marker M represented by the position in the marker placement area 6, and to move each marker M according to the evaluation. Therefore, when giving subjective evaluations to multiple evaluation targets, a relative evaluation based on a comparison between the evaluation targets can be easily performed.

[0041] When the information processing device 1 is reproducing evaluation target data of one marker M selected from the multiple markers M, the display mode of the one marker is changed to a display mode that makes the one marker distinguishable from the other markers M. In the example of FIG. 2, only the marker M4, whose evaluation target data is being reproduced, is displayed with white characters on a black background, and is displayed in a different manner from the other markers (M1 to M3, M5 to M7). The display modes may be distinguished in any manner, and may be distinguished, for example, by the color or shape of the marker M. This allows the evaluation to be carried out while clearly understanding which evaluation target data of the marker M is being reproduced.

[0042] A playback mode button 52 is disposed at the top of the evaluation input screen 5. In response to an operation of the playback mode button 52 by the evaluator, the information processing device 1 sets the operation mode related to the playback of the evaluation target data to a playback enabled mode in which the evaluation target data can be played back, or a playback inhibited mode in which the playback of the evaluation target data is inhibited. When the operation mode is set to the playback enabled mode, the information processing device 1 plays the sound of the marker M in response to the above-mentioned instruction of the evaluator (such as clicking or tapping on the marker M). On the other hand, when the operation mode is set to the playback inhibited mode, the information processing device 1 inhibits the playback of the sound of the marker M regardless of the instruction input by the evaluator. This makes it easier for the evaluator to concentrate on the evaluation, since it is possible to prevent unintentional playback of sound, for example, when it is desired to move the marker M without playing unnecessary sound.

[0043] The marker display setting area 7 is an area for inputting settings related to the display state of the marker M in the rectangular area 60. In the marker display setting area 7, numbers (1 to 7) representing the evaluation target data corresponding to the marker M are arranged vertically, and a check box 71 is arranged next to each of these numbers. The check boxes 71 next to the numbers are input elements for setting the display state of the marker M corresponding to the number.

[0044] The information processing device 1 makes the display state of the marker M whose check box 71 is not checked lighter than the display state of the marker M whose check box 71 is checked. In the example of FIG. 2, the marker M6 of the number "6" whose check box 71 is not checked is displayed lighter than the other number markers (M1 to M5, M7). In this way, by setting the display state of each marker M to a "normal display state" (first display state) or a "faint display state" (second display state) according to the instruction of the evaluator, visual confusion caused by all the markers M being displayed in the same display state at once can be reduced, and it is possible to make it easier to perform an evaluation focusing on a specific evaluation target. In addition, for evaluation targets other than the specific evaluation target of interest, the marker M can be visually recognized in a "faint display state", so that an overall comparative perspective is not lost.

[0045] The arrangement of numbers (1 to 7) of the markers M in the marker display setting area 7 represents the different hierarchical levels assigned to the markers M. That is, the higher a number is in the marker display setting area 7, the higher a hierarchical level the marker M corresponding to that number is assigned to. A display order of the markers M in the marker placement area 6 is determined for these multiple hierarchical levels, with the markers M in the higher hierarchical levels being determined to have a higher display order.

[0046] The information processing device 1 assigns a plurality of different hierarchical levels to a plurality of markers M in response to an instruction input by the evaluator. For example, the information processing device 1 displays a context menu 55 for a specific marker M (marker M4 in the example of FIG. 2) in response to an operation of the pointer 50 by the evaluator. The context menu 55 includes an instruction (to the front / to the front / to the back / to the back) for changing the hierarchy (display order) assigned to the specific marker M, as shown in FIG. 2, for example. The information processing device 1 changes the hierarchy (display order) assigned to the specific marker M in accordance with an instruction selected from the context menu 55 by operating the pointer 50. In this way, by setting the hierarchy of each marker M in response to an instruction from the evaluator, it is possible to make the specific marker M that the evaluator focuses on more easily visible in the marker arrangement area 6, even when a plurality of markers M are gathered in the same place.

[0047] The information processing device 1 may change the hierarchy of an arbitrary marker M in response to an operation of selecting a number of markers M equal to or greater than one in the marker display setting area 7 and moving the number in the up or down direction (eg, a drag operation using a mouse).

[0048] When two or more markers M are located at a location specified by an instruction input by an evaluator (e.g., a location specified by the pointer 50), the information processing device 1 selects the marker M assigned to the highest hierarchical level in the display order among the two or more markers, and moves the selected marker M. In this case, the information processing device 1 selects the marker M assigned to the highest hierarchical level in the display order among the two or more markers, and plays a sound of the selected marker M. Therefore, by setting the hierarchical level of an arbitrary marker M to a higher level, it becomes possible to easily select and move the marker M set to the higher hierarchical level or play a sound, even if another marker M is located in the same location as the marker M set to the higher hierarchical level.

[0049] The information processing device 1 may be configured so that the markers M whose check boxes 71 are not checked (markers M in the second display state) cannot be moved regardless of the instruction input by the evaluator. This allows the user to easily select and move only a specific marker M even when multiple markers M are located in the same place. Similarly, the information processing device 1 may suppress playback of the evaluation target data for the markers M whose check boxes 71 are not checked (markers M in the second display state) regardless of the instruction input by the evaluator. This allows easy selection of only a specific marker M and playback of the evaluation target data, even if multiple markers M are located at the same location.

[0050] In the marker display setting area 7, buttons 73 and 74 are arranged as input elements for selecting a marker M for playing back the evaluation target data. When the evaluator presses the button 73 or 74 (operates the input element), the information processing device 1 moves the hierarchical level of the marker M for playing back the evaluation target data up or down. This makes it possible to easily select a specific marker M and play back the evaluation target data, even when multiple markers M are located in the same place.

[0051] The information processing device 1 displays the number of the marker M that is playing the evaluation target data in the marker display setting area 7 in a manner different from the other numbers. In the example of Fig. 2, only the number "4" of the marker M4 that is playing sound is displayed in white letters on a black background, and is displayed in a manner different from the numbers of the other markers M. This makes it possible to more clearly grasp which marker M's evaluation target data is being played.

[0052] 2, a button 72 is disposed at the top of the marker display setting area 7 for inputting an instruction to switch between a state in which only the numbers of markers M whose checkboxes 71 are checked and a state in which the numbers of all markers M are displayed. Also, a scroll bar 75 is disposed at the right end of the marker display setting area 7 for inputting an instruction to scroll vertically the numbers of multiple markers M arranged vertically.

[0053] 3 is a diagram showing another example of the evaluation input screen 5. The evaluation input screen 5 shown in FIG. 3 includes elements similar to those of the evaluation input screen 5 shown in FIG. 2, and also includes a playback setting area 8A for inputting settings related to the repeated playback of the evaluation target. The playback setting area 8A includes radio buttons 81 for alternatively selecting a setting for not performing repeated playback of the evaluation target (playing only once), a setting for repeating playback of the evaluation target a number of times (number of repeated playbacks) specified by the evaluator, and a loop playback setting for repeating playback of the evaluation target without limiting the number of repeated playbacks. Furthermore, the playback setting area 8A includes an input field 82 for specifying the number of repeated playbacks when the setting for repeating playback of the evaluation target only for repeated playbacks is selected.

[0054] When the number of repeated playbacks is set in the playback setting area 8A, the information processing device 1 repeatedly plays back one evaluation target data the set number of repeated playbacks every time an instruction to play back one evaluation target data is input by the evaluator. This makes it easier for the evaluator to concentrate on the evaluation since the input operation for playing back the evaluation target can be omitted.

[0055] Furthermore, when loop playback (repeated playback of the evaluation target without limiting the number of times it is repeated) is set in the playback setting area 8A, the information processing device 1 repeatedly plays one evaluation target data without limiting the number of times it is repeated in response to an instruction from an evaluator who plays the one evaluation target data. This allows, for example, for an evaluation target that is difficult to evaluate, to repeat playback until the evaluator gives an instruction to end the loop playback, making it easier for the evaluator to concentrate on the evaluation.

[0056] FIG. 4 is a diagram showing another example of the evaluation input screen 5. The evaluation input screen 5 shown in FIG. 4 includes elements similar to those of the evaluation input screen 5 shown in FIG. 3, and also includes a playback setting area 8B for setting a playback order for two or more markers M. In the playback setting area 8B, a plurality of markers 83 for setting a playback order are arranged in the horizontal direction. One marker 83 corresponds to one marker M arranged in the marker arrangement area 6. One marker 83 corresponding to one marker M has the same number (number representing an evaluation target) as the one marker M. One marker 83 corresponding to one marker M is also arranged in the playback setting area 8B by, for example, an operation of moving the one marker M from the marker arrangement area 6 to the playback setting area 8B (e.g., a drag operation). The arrangement order of the plurality of markers 83 in the playback setting area 8B can be changed by moving each marker 83 by a drag operation or the like. The arrangement order of the multiple markers 83 represents the playback order of the multiple evaluation target data corresponding to the multiple markers 83, and in the example of Fig. 4, the marker 83 located on the left side has the earlier playback order. When the reset button 84 is pressed, the setting of the arrangement order of the markers 83 in the playback setting area 8B is reset.

[0057] When the playback order of two or more markers M is set in the playback setting area 8B, the information processing device 1 plays back the two or more evaluation target data corresponding to the two or more markers M in order according to the set playback order in response to an instruction from the evaluator to play back the two or more evaluation target data (e.g., an instruction to select one of the two or more markers M as a playback target). In this way, since the evaluation target data of two or more markers M can be played back in the order specified by the evaluator, it becomes easier to perform relative evaluation of two or more related evaluation targets. For example, when checking index values ​​given to two or more evaluation targets for one index, it becomes easier to determine whether the index value of each evaluation target is appropriate by playing back the two or more evaluation targets consecutively in the order according to the index value.

[0058] 5 to 8 are diagrams showing other examples of the evaluation input screen 5, and show examples of the evaluation input screen 5 on which a plurality of marker-related images P1 to P7 corresponding one-to-one to a plurality of markers are displayed. The marker-related images P1 to P7 (hereinafter sometimes referred to as "marker-related images P" without distinction) are images related to individual markers M, for example, images related to evaluation targets corresponding to individual markers M. In the examples of Figs. 5 to 8, the evaluation target is a sound generated by a passenger car (such as the sound of a door opening and closing), and the marker-related image P is an image representing the passenger car.

[0059] The evaluation input screen 5 shown in Fig. 5 is the same as the evaluation input screen 5 shown in Fig. 2, except that a marker-related image P is arranged instead of the marker M (a round shape with the number of the evaluation target placed in the center). This allows the marker-related image P to provide more information than just an identification label for the evaluation target, and therefore allows the evaluator to have a more concrete and intuitive image of the evaluation target.

[0060] 5, the numbers (1 to 7) of each marker M arranged in the marker display setting area 7 on the evaluation input screen 5 shown in Fig. 2 are replaced with images 77 that are the same as the marker-related images P corresponding to each marker M. This makes it possible to set the hierarchy and display state for the marker-related images P in the same way as for the markers M on the evaluation input screen 5 shown in Fig. 2.

[0061] The evaluation input screen 5 shown in FIG. 6 is obtained by arranging images 77 (same as the marker-related images) corresponding to the markers M one-to-one next to the numbers (1 to 7) of the markers M arranged in the marker display setting area 7 in the evaluation input screen 5 shown in FIG. 2. Each image 77 corresponding to each marker M has a display mode associated with each marker M by being arranged next to the numbers (1 to 7) of each marker M. Also, each image 77 has a display mode distinguishable from other markers M and other images 77. That is, in the example of FIG. 6, the multiple images 77 (same as the marker-related images) arranged in the marker display setting area 7 are displayed in association with the multiple markers M arranged in the marker arrangement area 6. This makes it possible to recognize the images 77 arranged in the marker display setting area 7 and the markers M arranged in the marker arrangement area 6 in association with each other. Therefore, similar to the evaluation input screen 5 shown in FIG. 5, it is possible to give the evaluator a concrete and intuitive image of the evaluation target.

[0062] 7 is an evaluation input screen 5 shown in Fig. 2, in which a marker-related image P corresponding to one marker M selected from a plurality of markers M is displayed in a speech bubble in response to an instruction input by the evaluator (e.g., an instruction to select the marker M by operating the pointer 50). This allows the evaluator to give a concrete and intuitive image of the evaluation target arbitrarily selected by the evaluator through the marker-related image P.

[0063] 8 is an evaluation input screen 5 shown in FIG. 2, in which a marker-related image P corresponding to one marker M selected from a plurality of markers M is displayed in an area 56 outside the marker placement area 6 in response to an instruction input by an evaluator (e.g., an instruction to select marker M by operating pointer 50). In area 56, a figure identical to the marker M (a round figure with the number of the evaluation target placed in the center) is arranged together with the marker-related image P corresponding to the marker M. As a result, similar to the evaluation input screen 5 shown in FIG. 7, the marker-related image P can give a concrete and intuitive image of the evaluation target arbitrarily selected by the evaluator.

[0064] FIG. 9 is a diagram showing an example of a screen 90 for setting a playback target section when playing back an evaluation target. The information processing device 1 sets a part of the entire playback section in one evaluation target data as a playback target section in response to an instruction input by an evaluator. The screen 90 is a screen displayed when inputting the setting of the playback target section. In the example of FIG. 9, the screen 90 has a waveform diagram 91 showing the waveform of the sound of the evaluation target, a seek bar 92 showing the playback status of the sound of the evaluation target, pointers 93 and 94 for setting the start time and end time of the playback target section, respectively, buttons 95, 96, and 97 for controlling the start, pause, and stop of the sound playback, respectively, a button 98 for saving the playback target section set by the pointers 93 and 94, and a button 99 for returning to the evaluation input screen 5 without saving the setting of the playback target section.

[0065] When a playback target section is set for one evaluation target data on the above-mentioned screen 90 (FIG. 9), the information processing device 1 plays back the playback target section in the one evaluation target data in response to an instruction from an evaluator to play back the one evaluation target data (e.g., an instruction to select the marker M by operating the pointer 50). This makes it easier to perform a detailed evaluation of a specific portion of the entire playback section of the evaluation target.

[0066] 10 is a flowchart for explaining an example of a process for inputting various settings. The information processing device 1 repeatedly executes the process shown in the flowchart of FIG.

[0067] The information processing device 1 checks whether an instruction from the evaluator to set the marker placement area 6 has been input (ST100), and if an instruction has been input (Yes in ST100), sets the marker placement area 6 in accordance with the instruction (ST105). For example, if an instruction requesting customization of an index guide element or the like in the marker placement area 6 is input, the information processing device 1 changes the display of the marker placement area 6 in accordance with the instruction.

[0068] The information processing device 1 checks whether an instruction to change the setting related to the display state of the marker M (marker display state setting) has been input in the marker display setting area 7 of the evaluation input screen 5 (ST110), and if an instruction has been input (Yes in ST110), it saves the marker display state setting changed in response to the instruction (ST115). For example, when the display state or hierarchy of the marker M is changed in the marker display setting area 7, the information processing device 1 saves the marker display state setting that reflects the changed display state and hierarchy.

[0069] The information processing device 1 checks whether an instruction to set a marker-related image for evaluation has been input (ST120), and if an instruction has been input (Yes in ST120), sets the display mode of the marker-related image according to the instruction (ST125). For example, if an instruction to select one of the display modes of the marker-related image shown in Fig. 5 to Fig. 8 is input, the information processing device 1 displays the marker-related image on the evaluation input screen 5 in the selected display mode.

[0070] The information processing device 1 checks whether an instruction to change the operating mode has been input using the playback mode button 52 (ST130), and if an instruction has been input (Yes in ST130), it sets the operating mode to a "playback mode" or a "playback suppression mode" in accordance with the instruction (ST135).

[0071] The information processing device 1 checks whether an instruction from the evaluator to input settings regarding the playback of the evaluation target has been input (ST140), and if an instruction has been input (Yes in ST140), sets the playback of the evaluation target according to the instruction (ST145). For example, when a setting for the number of repetitions of playback or loop playback is input in the playback setting area 8A (FIG. 3), the information processing device 1 repeatedly plays back the evaluation target according to the setting. Also, when a setting for the playback order of markers M is input in the playback setting area 8B, the information processing device 1 plays back the markers M in order according to the set playback order. Furthermore, when a playback target section is set on the screen 90 (FIG. 9), the information processing device 1 plays back the set playback target section.

[0072] 11 to 14 are flowcharts for explaining an example of a process for inputting an evaluation of each evaluation target on the evaluation input screen 5. While performing the evaluation input process, the information processing device 1 repeatedly executes the processes shown in the flowcharts of FIGS.

[0073] First, the information processing device 1 displays, on the evaluation input screen 5, a marker arrangement area 6 in which a plurality of markers M are arranged in one-to-one correspondence with a plurality of evaluation target data (ST200). When a setting for a mark-related image is input in step ST125 shown in Fig. 10, the information processing device 1 displays the mark-related image on the evaluation input screen 5 in a display mode according to the setting.

[0074] Next, the information processing device 1 checks whether an instruction to select a specific marker M (an instruction to select the marker M with the pointer 50, an instruction to select the marker M with the buttons 73 and 74 in the marker display setting area 7) has been input (ST205). If an instruction to select a specific marker M has not been input (No in ST205), the information processing device 1 proceeds to step ST255, which will be described later.

[0075] When an instruction to select a specific marker M is input (Yes in ST205), the information processing device 1 checks whether multiple markers M are located at the same location as the specific marker M (ST210). When multiple markers M are located at the same location (Yes in ST210), the information processing device 1 selects the marker M that is at the highest level in the hierarchy (display order) among the multiple markers M as the specific marker (ST220). On the other hand, when no other marker M is located at the same location as the specific marker M (No in ST210), the information processing device 1 selects the specified specific marker M (ST215). Also, when one marker M is selected by the buttons 73 and 74 in the marker display setting area 7, the information processing device 1 selects the one marker M as the specific marker M (ST215).

[0076] When the information processing device 1 selects a specific marker M in step ST215 or ST220, it checks whether the evaluation target of another marker M is being played back (ST225). If the evaluation target of another marker M is being played back (Yes in ST225), the information processing device 1 stops the playback (ST230).

[0077] In addition, when the information processing device 1 selects a specific marker M in step ST215 or ST220, it checks whether the operating mode is a playback mode (ST235), and if it is not a playback mode (playback suppression mode) (No in ST235), it proceeds to step ST255 described later.

[0078] Furthermore, when the information processing device 1 selects a specific marker M in step ST215 or ST220, it checks whether the specific marker M is being played back (ST240), and if the specific marker M is being played back (Yes in ST240), it proceeds to step ST255, which will be described later.

[0079] If the operating mode is a playback mode (Yes in ST235) and the selected specific marker M is not being played (No in ST240), the information processing device 1 starts playing the evaluation target data corresponding to the specific marker M (ST245).

[0080] If a playback target section is set for the evaluation target data of a specific marker M (ST145 in FIG. 10), the information processing device 1 plays the evaluation target data from the start time of the set playback target section. If a playback order is set for a specific marker M (ST145 in FIG. 10), the information processing device 1 starts playing the first marker M in the playback order. Furthermore, if a repeat playback count for a specific marker M is set (ST145 in FIG. 10), the information processing device 1 sets the count value of the number of repeated playbacks to an initial value (first time).

[0081] When proceeding to step ST255, the information processing device 1 checks whether a specific marker M is selected. If a specific marker M is selected (Yes in ST255), the information processing device 1 checks whether an instruction to move the specific marker M (for example, an instruction to drag the marker M by operating the pointer 50) has been input (ST260). If an instruction to move the specific marker M has been input (Yes in ST260), the information processing device 1 moves the specific marker M in accordance with the input instruction (ST265).

[0082] The information processing device 1 also checks whether an instruction to cancel the selection of the marker M (for example, an instruction to select a location in the marker placement area 6 where the marker M is not present with the pointer 50) has been input (ST270). If an instruction to cancel the selection has not been input (No in ST270), the information processing device 1 proceeds to step ST285, which will be described later.

[0083] When an instruction to cancel the selection of the marker M is input (Yes in ST270), if the marker M to be canceled is being played back (Yes in ST275), the information processing device 1 stops the playback (ST280).

[0084] When proceeding to step ST285, the information processing device 1 checks whether or not the specific marker M is being played back. If the specific marker M is not being played back (No in ST285), the information processing device 1 proceeds to step ST345, which will be described later.

[0085] If a specific marker M is being played back (Yes in ST285), the information processing device 1 checks whether the playback time (playback position) has reached the end of the playback range (ST290). If the playback position has not reached the end of the playback range (No in ST290), the information processing device 1 proceeds to step ST345, which will be described later.

[0086] If the playback position of the marker M being played back has reached the end of the playback range (Yes in ST290), the information processing device 1 adds 1 to the total number of times that playback has been performed for this marker M (ST295).

[0087] Furthermore, if the playback position of the marker M being played back has reached the end of the playback range (Yes in ST290), the information processing device 1 checks whether a playback order is set for this marker M (ST300). If a playback order is not set (No in ST300), the information processing device 1 checks whether loop playback is set in the playback setting area 8A (FIGS. 3 and 4) (ST320). If loop playback is set (Yes in ST340), the information processing device 1 starts playback from the beginning of the playback range for this marker M (ST340) and proceeds to step ST345, which will be described later. If loop playback is not set in the playback setting area 8A (FIGS. 3 and 4) (No in ST320), the information processing device 1 adds 1 to the count value of the number of repeated playbacks (ST325), and checks whether the count value after the increment exceeds the limit of the number of repeated playbacks set in the playback setting area 8A (ST330). If the count value of the number of repeated playbacks exceeds the set limit on the number of repeated playbacks (Yes in ST330), the information processing device 1 stops playback of that marker M (ST335). If the count value of the number of repeated playbacks does not exceed the limit (No in ST330), the information processing device 1 starts playback of this marker M from the beginning of the playback range (ST340), and proceeds to step ST345, which will be described later.

[0088] If the playback position of the marker M being played back has reached the end of the playback range (Yes in ST290) and a playback order has been set for this marker M (Yes in ST300), the information processing device 1 checks whether playback has been completed up to the last marker M in the playback order (ST305). If playback has not been completed up to the last marker M in the playback order (No in ST305), the information processing device 1 selects the next marker M in the playback order (ST315), starts playback of the selected marker M from the beginning of the playback range (ST340), and proceeds to step ST345, which will be described later. If playback has been completed up to the last marker M in the playback order (Yes in ST305), the information processing device 1 selects the first marker M in the playback order (ST310), performs the same processes as those in steps ST320 to ST340 described above for the selected marker M, and proceeds to step ST345, which will be described later.

[0089] Proceeding to step ST345, the information processing device 1 checks whether an instruction to end the input of the evaluation (an instruction to press the button 53) has been input by the evaluator. If an instruction to end the input of the evaluation has been input (Yes in ST345), the information processing device 1 acquires the position of each marker M in the marker placement area 6 as the evaluation result (ST350). The information processing device 1 also acquires the number of movements of each marker M, the movement distance of each marker M, the movement time of each marker M, information on the trajectory of the movement of each marker, the number of floors on which each marker M was played, and the like, as information on the process of the evaluation by the evaluator (evaluation process information) (ST355). Then, the information processing device 1 transmits the evaluation result of each marker M (the position of each marker) and the evaluation process information to the server device 2 (ST360).

[0090] As described above, according to the present embodiment, the sound of the marker M arranged in the marker arrangement area 6 is played in response to the evaluator's instruction on the evaluation input screen 5. In addition, the "position-index relationship" in the marker arrangement area 6 is guided to the evaluator by the index guide element, and the marker M is moved in response to the instruction of the evaluator who recognizes this guidance. Then, as the evaluation result for each evaluation target, the position of each marker M in the marker arrangement area 6, or the value of the index of each evaluation target based on the position of each marker M and the "position-index relationship" is acquired. This allows the evaluator to evaluate the sound of each marker M while comparing the index values ​​of each marker M represented by the position in the marker arrangement area 6, and to move each marker M according to the evaluation. Therefore, when subjective evaluations are given to multiple evaluation targets, a relative evaluation based on a comparison between the evaluation targets can be easily performed. By performing a relative evaluation, the evaluator can give a more subjectively correct evaluation to multiple evaluation targets.

[0091] (Modification) Next, some variations of the above-described embodiment will be described.

[0092] Fig. 15 is a diagram showing an example of a marker placement area when there is one evaluation item. The marker placement area 6 shown in Fig. 2 etc. is a two-dimensional planar area, and two coordinate values ​​(horizontal position, vertical position) indicating a position on the two-dimensional plane are associated with index values ​​of two evaluation items. On the other hand, when there is one index for an evaluation item, as shown in Fig. 15, a linear area may be used as the marker placement area. In this case, the index guide element may be something that guides the relationship between one coordinate value indicating the position on the line on which the marker is placed and one index value (for example, an antonym placed on both ends of the line).

[0093] Fig. 16 is a diagram showing an example of a marker placement area when there are multiple evaluation items. When there are multiple indicators for the evaluation items, multiple linear marker placement areas (Fig. 15) may be arranged in one direction as shown in Fig. 16.

[0094] Fig. 17 is a diagram showing another example of the marker placement area when there are multiple evaluation items. When there are multiple indexes for the evaluation items, multiple linear marker placement areas (Fig. 15) may be radially placed as shown in Fig. 17. In this case, as shown in Fig. 17, by connecting markers M corresponding to the same evaluation target with a common straight line, it becomes easier to distinguish the evaluation results of each marker M.

[0095] In addition to these examples, when there are three evaluation items, the marker placement area may be a three-dimensional space. In this case, the three coordinate values ​​that represent the position of the marker M in space may be associated with the index values ​​of the three evaluation items, respectively. The index guide element may also be something that guides the relationship between the three coordinate values ​​that represent the spatial position where the marker is placed and the value of one index (antonyms placed at both ends of the coordinate axis).

[0096] 18A to 18H are diagrams showing examples of setting the index guide element. In step ST100 described above, the information processing device 1 may customize the index guide element in the marker placement area as shown in FIG. 18A to 18H in response to an instruction from the evaluator. That is, the information processing device 1 may set a word (one word or two antonyms) representing the index in the index guide element, set the presence or absence of one or more scales representing the value of the index in the index guide element, set the interval between one or more scales, and set one or more scale labels to be attached to one or more scales in response to an instruction from the evaluator. However, it is desirable that the contents of the settings that the evaluator can freely customize are limited to a range that does not affect the evaluation result of the evaluator.

[0097] FIG. 18A shows an example of a marker placement area in which scale marks are not displayed. FIG. 18B shows an example of a marker placement area in which tick marks are displayed but tick mark labels are not displayed. FIG. 18C shows an example of a marker arrangement area in which a numerical value indicating an index value is added to each scale mark as a scale label. FIG. 18D shows an example of a marker placement area in which the numerical value of the scale label attached to the center scale mark is "0," and the numerical values ​​of the scale labels increase toward the left and right. Fig. 18E shows an example of a marker placement area using words expressing degree, such as "slightly," "quite a bit," and "neither," as scale labels. In the example of Fig. 18E, the scale is divided into five stages.

[0098] Figures 18F and 18G show examples of marker placement areas in which only one indicator word is used in the indicator guide element. In the example of Figure 18F, the indicator word "Strong" is placed at the right edge of the marker placement area, and in the example of Figure 18G, the indicator word "Level of Strength" is placed at the bottom of the marker placement area. Having a single term for the metrics simplifies visualization and makes interpretation of the evaluation results more intuitive, which is especially useful when non-technical evaluators are evaluating products. Using a single term to describe a metric also helps focus and improve understanding of a particular criterion, which is useful when you want to emphasize a particular improvement goal or criterion. Furthermore, by using a single term to represent the indicator, the relative position of each piece of data in relation to that term can be seen at a glance, which is useful when there are many evaluation targets to compare or when you want to grasp the overall picture. Using a single word to describe the indicator also avoids negative connotations, which can be useful when certain raters are sensitive to negative feedback or during presentations.

[0099] FIG. 18H shows an example of a marker placement area in which the position of the marker M of the reference sound source is fixed. This allows the sound of other markers M to be evaluated based on a comparison with the marker M of the reference sound source. In particular, when the sign guide element has only one word that represents an index, placing the marker M of the reference sound source in a fixed location can prevent the evaluation from becoming vague.

[0100] 19A to 19C and 20 are diagrams showing examples of graphs representing the evaluation results of each marker M. The server device 2 collects evaluation results from the information processing devices 1 of a large number of evaluators and analyzes the collected evaluation results to generate graphs such as those shown in Figs. 19A to 19C and 20. The information processing device 1 specifies the contents and conditions of analysis of the evaluation results to the server device 2, and displays the analysis results (graphs, etc.) obtained by the server device 2 on the display unit 13.

[0101] 19A is a histogram based on the results of evaluations of one evaluation item ("strong / weak") by multiple evaluators. The server device 2 generates a histogram as shown in FIG. 19A by dividing the index values ​​of the evaluation results by each evaluator into a plurality of predetermined index categories and counting the frequency of the evaluation results divided into each index category.

[0102] In the histogram of FIG. 19A, there is one evaluation item, but when there are two evaluation items, the frequency of the evaluation results may be displayed on a two-dimensional grid. For example, a predetermined square may be placed at each of the intersections of two index categories on the two-dimensional grid, and the frequency of the evaluation results belonging to the two index categories corresponding to the squares may be displayed by the color of the square or other display manner (such as the size of the square). Also, when there are three or more evaluation items, for example, two evaluation items may be arbitrarily selected, and the other evaluation items may be fixed to arbitrary index values, and a graph using the above-mentioned two-dimensional grid for the two evaluation items may be generated.

[0103] The histogram in Figure 19A shows the index category with the highest evaluation result and the average index value. This makes it possible to clearly show the concentration tendency of the data and the extreme values. This is important information for understanding how scattered or concentrated the evaluation of the evaluation target is.

[0104] In addition, the histogram shown in FIG. 19A shows a mark representing the average value of the index values ​​obtained by the same analysis for other evaluation targets (a triangular mark representing the average value of the other evaluation targets "Company A" and "Company B"). This allows intuitive visualization of how superior or inferior one evaluation target is compared to other evaluation measures. By visualizing the comparison between one evaluation target and other evaluation targets, for example, the competitiveness and goal achievement level of a product related to that evaluation target can be understood at a glance, making it easier to plan necessary improvement measures. In addition, by visualizing the comparison of evaluation results, the evaluation results can be understood more deeply than when the evaluation results are conveyed only by numbers and words, and the dialogue between the parties involved can be promoted more concretely and efficiently.

[0105] Such analyses play a central role in the feedback loop of the development of the product involved in the evaluation. Visualizing the evaluation results using histograms can provide concrete directions for product improvement, thus contributing to the improvement of the product quality.

[0106] FIG. 19B is a two-axis scatter diagram based on the results of evaluations of one evaluation object by multiple evaluators on two evaluation items ("strong / weak" and "big / small"). In this two-axis scatter diagram, the individual evaluation results by multiple evaluators are represented by one marker on a two-dimensional plane. By setting the display mode (color, shape, etc.) of the marker according to the attributes of the evaluators and the type of evaluation object, it is possible to grasp at a glance the tendency of evaluation according to the attributes of the evaluators and the type of evaluation object.

[0107] FIG. 19C is a three-axis scatter diagram based on the results of evaluations of one evaluation object by multiple evaluators on three evaluation items ("strong / weak," "big / small," and "like / dislike"). In a three-axis scatter diagram, the individual evaluation results by multiple evaluators are represented by one marker located in three-dimensional space. In a three-axis scatter diagram, the display mode of the marker may also be set according to the attributes of the evaluators and the type of evaluation object. The graph of the three-axis scatter diagram may be made freely rotatable by mouse operation, etc. In addition, in a three-axis scatter diagram as shown in Fig. 19C, one or more planes parallel to the two coordinate axes may be arranged, and markers of the evaluation results may be projected onto the planes. By displaying a two-axis scatter diagram in addition to a normal three-axis scatter diagram, it becomes easier to visually grasp the trend of the evaluation.

[0108] Figure 20 shows a parallel graph based on the results of evaluations of one evaluation target by multiple evaluators on four evaluation items (strong / weak, big / small, like / dislike, clear / muddy). In a parallel graph, axes representing the index values ​​of each evaluation item are arranged in parallel, and one point on the axis represents one evaluation result by one evaluation target. Points representing evaluation results by the same evaluator on multiple axes are connected by a common line. In a parallel graph, evaluation results for multiple evaluation items by the same evaluator are represented by one line, so the overall evaluation tendency of the evaluator can be visually grasped by the tendency of the line to bend or the tendency of the lines to be dense. By setting the color of the line according to the attributes of the evaluator and the type of evaluation target, the evaluation tendency according to the attributes of the evaluator and the type of evaluation target can be visually grasped.

[0109] In addition, in a parallel graph such as that shown in Figure 20, the lines to be displayed can be narrowed down according to the range of indicators on each axis, the attributes of the evaluator, the type of evaluation target, etc., making it possible to visually grasp the evaluation trends from various perspectives.

[0110] In addition, when there are more than two evaluation items, for example, two evaluation items may be arbitrarily selected and the other evaluation items may be fixed to arbitrary index values, and the above-mentioned two-axis scatter plot for the two evaluation items may be generated.

[0111] The server device 2 may analyze evaluation process information (such as the number of times the evaluation target is played, the number of times the marker is moved, etc.) in addition to the above-mentioned analysis of the evaluation result.

[0112] It is presumed that the more the number of times the evaluation target is played or the number of times the marker is moved, the more information the evaluator gathers about the evaluation target and the more accurate the evaluation. In addition, the number of times the evaluation target is played or the number of times the marker is moved are indicators of how actively the evaluator is involved in the evaluation process. If an evaluator plays the evaluation target many times and moves the marker frequently, the evaluator is likely to be serious about the evaluation. In addition, by analyzing the number of times the evaluation target is played or the number of times the marker is moved, it is possible to understand how much time and effort the evaluator spends on the evaluation. This makes it possible to evaluate the efficiency of the evaluation process. For example, when evaluating the same number of evaluation targets, if some evaluators play the targets more times and move the marker more times than other evaluators, the evaluators may spend more time and effort on the evaluation process. Considering the above advantages, analyzing the number of times the evaluation target is played or the number of times the marker is moved is a useful means of evaluating the reliability and involvement of the evaluators in their evaluations and the efficiency of the evaluation process.

[0113] In the above-described example of the present embodiment, no reference is made to the initial position of the marker for each evaluation target in the marker placement area, but when evaluating multiple indices in particular, in the initial state in which the marker positions have not yet been determined at all, the evaluation method of replacing the values ​​of multiple indices with the marker positions (moving the marker positions according to the evaluation results) may be difficult for the evaluator. Therefore, in this embodiment, the initial positions of the markers for each evaluation target in the marker placement area may be set by a method that does not involve the operation (movement) of the markers.

[0114] For example, the information processing device may set the initial position of the marker of each evaluation target in the marker placement area to a predetermined position. In this case, for example, the information processing device may set a fixed position predetermined for each evaluation target as the initial position of the marker, or may randomly determine the initial position of the marker of each evaluation target for each evaluator or each evaluation. In addition, when one evaluator has previously performed evaluation on the same evaluation target, the initial position of the marker of the evaluation target may be set based on the past evaluation results (such as past evaluation results collected in the server device 2). When multiple evaluators have performed evaluation on the same evaluation target, the initial value of the marker of the evaluation target may be set based on the average value of the evaluation results (e.g., index values) by the multiple evaluators.

[0115] In addition, the information processing device may acquire the results of an evaluation (e.g., index values) separately performed by an evaluator for each evaluation target in order to set the initial position of each marker in the marker placement area. In this case, for example, the information processing device may set the initial position of each marker based on the acquired evaluation results of each evaluation target.

[0116] Furthermore, the information processing device may have the evaluator evaluate each evaluation object (hereinafter, may be referred to as "initial evaluation") in order to set the initial position of each marker in the marker placement area. For example, in the process of setting the initial position of each marker in the marker placement area, the information processing device may sequentially play back each evaluation object using a playback device (such as the sound output unit 14) in response to an instruction input by the evaluator, obtain a result of the initial evaluation (e.g., an index value) given by the evaluator for each played evaluation object, and determine the initial position of each marker based on the obtained result of the initial evaluation of each evaluation object. In this case, the information processing device may input the result of the initial evaluation for each reproduced evaluation object by a method other than the operation (movement) of a marker. For example, the information processing device may input the result of the initial evaluation by a method of having the evaluator select one index value from a predetermined number of index values ​​using a radio button, a pull-down list, etc., or by having the evaluator directly enter an index value (numerical value). In this case, the information processing device may play back the evaluation objects for the initial evaluation in a predetermined order, or may be able to change the order of playback of the evaluation objects in response to instructions input by the evaluator. Furthermore, in this case, the information processing device may determine initial positions of markers for all evaluation objects based on the results of the initial evaluation, and then display the markers of all evaluation objects with their initial positions reflected in the marker placement area, or each time it determines the initial position of one marker for one evaluation object based on the results of the initial evaluation, it may display that one marker with its initial position reflected in the marker placement area.

[0117] The present invention is not limited to the above-described embodiments. Any embodiment or application thereof in which a person skilled in the art appropriately adds or removes components or modifies the design, or any combination of features of the embodiments is included within the scope of the present invention as long as it includes the gist of the present invention. For example, the evaluation target in this embodiment is not limited to sound, still images, and videos, and may be something that stimulates at least one of the senses such as touch, smell, taste, etc. In this case, the playback device may be a device that reproduces at least one of the senses such as touch, smell, taste, etc., and the evaluation target data may be data used to reproduce a desired sense in such a device.

[0118] Below, supplementary notes related to this embodiment will be described. [1] A method for supporting an evaluator in providing a subjective evaluation to each of a plurality of evaluation targets, comprising: the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; a step of displaying, by the information processing device, a marker arrangement area in which a plurality of markers corresponding one-to-one to a plurality of the evaluation target data are arranged on a display device; a step of moving one of the markers selected from the plurality of markers in response to an instruction input by the evaluator by the information processing device; a step of the information processing device reproducing, in the reproduction device, one of the evaluation target data corresponding to one of the markers selected from the plurality of markers in response to an instruction input by the evaluator; The information processing device acquires evaluation results for the plurality of evaluation targets. having the step of displaying the marker placement area includes displaying the marker placement area including one or more index guide elements that guide a predetermined relationship between a position of the marker in the marker placement area and a value of the index for each of one or more indexes related to the evaluation of the evaluation object; The step of acquiring the evaluation result for one of the evaluation targets includes: acquiring positions of the markers in the marker placement area; Or, acquiring the positions of the plurality of markers in the marker arrangement area, and acquiring one or more values ​​of the index for each of the plurality of evaluation objects based on the acquired positions of the markers and the predetermined relationship. Including, method. [2] the information processing device, when reproducing the evaluation target data corresponding to one of the markers selected from the plurality of markers, changes a display mode of the one marker to a display mode distinguishable from the other markers; The method described in [1]. [3] the step of reproducing the evaluation target data includes reproducing one of the evaluation target data corresponding to one marker at least one of when one of the markers is selected in response to an instruction input by the evaluator and when the one marker is moved in response to an instruction input by the evaluator; The method described in [2]. [4] the step of reproducing the evaluation target data includes, when the evaluation target data corresponding to one of the markers selected from the plurality of markers is reproduced, when an instruction from the evaluator to deselect the one marker is input, stopping the reproduction of the evaluation target data; The method described in [3]. [5] The information processing device includes a step of setting an operation mode related to the reproduction of the evaluation target data to a reproduction enabled mode in which the evaluation target data can be reproduced or a reproduction inhibiting mode in which the reproduction of the evaluation target data is inhibited in response to an instruction input by the evaluator, the step of reproducing the evaluation target data includes, when the reproduction inhibition mode is set, inhibiting reproduction of the evaluation target data regardless of an instruction input by the evaluator; The method described in [3]. [6] the information processing device includes a step of setting the number of times the evaluation target data is to be repeatedly played back in response to an instruction input by the evaluator; the step of reproducing the evaluation target data includes repeatedly reproducing the one evaluation target data a set number of times each time an instruction to reproduce the one evaluation target data is input by the evaluator, The method described in [3]. [7] the step of setting the number of repeated playbacks includes setting a loop playback in which the evaluation target data is repeatedly played back without limiting the number of repeated playbacks in response to an instruction input by the evaluator; the step of playing the evaluation target data includes, when the loop playback is set, repeatedly playing the one evaluation target data in response to an instruction from the evaluator to play the one evaluation target data, without limiting the number of times the evaluation target data is repeatedly played; The method described in [6]. [8] the information processing device includes a step of setting a playback order for two or more of the markers selected from a plurality of the markers in response to an instruction input by the evaluator; the step of playing back the evaluation target data includes, when the playback order is set for two or more of the markers, playing back the two or more evaluation target data in an order according to the playback order in response to an instruction from the evaluator to play back the two or more evaluation target data corresponding to the two or more markers; The method described in [2]. [9] the step of setting the playback order for the two or more markers includes setting a loop playback for repeatedly playing back the two or more evaluation target data corresponding to the two or more markers in response to an instruction input by the evaluator; the step of playing back the evaluation target data includes, when the loop playback is set for two or more of the markers for which the playback order is set, repeatedly playing back the two or more evaluation target data in an order corresponding to the playback order; The method described in [8].

[10] the information processing device includes a step of setting a part of the entire playback section in one of the evaluation target data as a playback target section in response to an instruction input by the evaluator; the step of reproducing the evaluation target data includes, when the reproduction target section is set for one of the evaluation target data, reproducing the reproduction target section in the one of the evaluation target data in response to an instruction from the evaluator to reproduce the one of the evaluation target data; The method described in [2].

[11] the step of setting the playback target section of one of the evaluation target data includes setting loop playback for repeatedly playing back the playback target section of the one of the evaluation target data in response to an instruction input by the evaluator; the step of playing the evaluation target data includes, when the loop playback is set for one of the evaluation target data in which the playback target section is set, repeatedly playing the playback target section of the one of the evaluation target data; The method described in

[10] .

[12] The step of displaying the marker placement area includes: the index guide element providing guidance on the relationship between a value of one coordinate representing a position on a line on which the marker is placed and a value of one of the indices; The index guide element provides guidance on the relationship between two coordinate values ​​that indicate a position on a plane on which the marker is placed and the values ​​of the two indices; the index guide element for guiding the relationship between three coordinate values ​​representing a spatial position where the marker is placed and the values ​​of the three indices; displaying the marker placement area including at least one of The method described in [1].

[13] The step of displaying the marker placement area includes a step of setting the index guide element in response to an instruction input by the evaluator, setting one word or two antonyms representing the indicator in the indicator guide element; Setting the presence or absence of one or more scales representing the value of the index in the index guide element and / or the interval of the one or more scales; setting one or more scale labels to be attached to one or more scales representing the value of the index in the index guide element; The method includes at least one of the steps of: The method described in [1].

[14] a plurality of marker-related images in one-to-one correspondence with a plurality of said markers; A step in which the information processing device displays the marker-related image in the marker arrangement area, replacing each of the plurality of markers with a corresponding marker-related image in response to an instruction input by the evaluator; displaying the marker-related image corresponding to one of the markers selected from the plurality of markers in response to an instruction input by the evaluator; displaying a plurality of the markers and a plurality of the marker-related images in association with each other in a display manner in which one of the markers and one of the marker-related images in a corresponding relationship have a display manner in which they are related to each other and are distinguishable from other markers and other marker-related images in response to an instruction input by the evaluator; The method includes at least one of the steps of: The method described in [1].

[15] the information processing device sets a display state of each of the markers to a first display state or a second display state that is lighter than the first display state in response to an instruction input by the evaluator; the step of displaying the marker placement area includes displaying each of the markers in the first display state or the second display state in accordance with the setting of the display state. The method described in [1].

[16] the information processing device includes a step of assigning a plurality of different hierarchies to a plurality of the markers in response to an instruction input by the evaluator; a display order in the marker placement area is determined for the plurality of hierarchical layers; the step of displaying the marker placement area includes, when two or more of the markers are located at the same location in the marker placement area, displaying, at the location, the marker to which the hierarchical level with the highest display rank is assigned among the two or more markers; The method described in

[15] .

[17] the step of moving the marker includes, when two or more of the markers are located at a location specified by an instruction input by the evaluator, selecting the marker to which the hierarchical level with the highest display rank is assigned among the two or more markers, and moving the selected marker; the step of reproducing the evaluation target data includes, when two or more of the markers are located at a location specified by an instruction input by the evaluator, selecting the marker to which the hierarchical level with the highest display rank is assigned from among the two or more markers, and reproducing the evaluation target data for the selected marker; The method described in

[16] .

[18] The step of moving the marker includes, as evaluation process information relating to the process of evaluation by the evaluator, The number of movements of each of the markers; The distance traveled by each of the markers; and the movement time of each of the markers; Information regarding the trajectory of movement of each of the markers; obtaining at least one of The method described in [1].

[19] the step of reproducing the evaluation target data includes acquiring the number of reproductions of each of the markers as evaluation process information relating to the process of evaluation by the evaluator; The method described in [1].

[20] A step of setting an initial position of each of the markers in the marker placement area, reproducing each of the evaluation objects in sequence by the reproduction device in response to an instruction input by the evaluator; obtaining a result of an initial evaluation given by the evaluator for each of the reproduced evaluation objects; determining an initial position of each of the markers based on the obtained result of the initial evaluation of each of the evaluation objects; The method includes the steps of: the step of displaying the marker placement area includes displaying, on the display device, the marker placement area in which each of the markers is placed at the initial position. The method described in [1]. [twenty one] A program including instructions for causing an information processing device to perform a process for supporting an evaluator in providing a subjective evaluation to each of a plurality of evaluation targets, the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; The processing performed by the information processing device in accordance with the instructions includes each step of the method described in any one of [1] to

[20] . program. [twenty two] An information processing device that supports an evaluator in giving a subjective evaluation to each of a plurality of evaluation targets, comprising: the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; A processing section; a memory unit that stores instructions to be executed by the processing unit; The processing performed by the processing unit in accordance with the instructions includes each step of the method described in any one of [1] to

[20] . Information processing device. [twenty three] An information processing device that supports an evaluator in giving a subjective evaluation to each of a plurality of evaluation targets, comprising: the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; [1] to

[20] , comprising a means for carrying out each step of the method. Information processing device. [twenty four] A system for supporting an evaluator in giving a subjective evaluation to each of a plurality of evaluation targets, comprising: the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; One or more information processing devices; A server device, The information processing device has a means for performing each step of the method described in any one of [1] to

[20] , the server device collects the evaluation results obtained in the step of acquiring evaluation results for the plurality of evaluation targets from one or more of the information processing devices; system. [Explanation of symbols]

[0119] 1...information processing device, 11...communication unit, 12...input unit, 13...display unit, 14...sound output unit, 15...storage unit, 151...program, 16...processing unit, 2...server device, 5...screen for inputting evaluation, 6...marker placement area, 7...marker display setting area, 8A, 8B...playback setting area, 9...communication network

Claims

1. A method for supporting an evaluator in providing a subjective evaluation to each of a plurality of evaluation targets, comprising: the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; a step of displaying, by the information processing device, a marker arrangement area in which a plurality of markers corresponding one-to-one to a plurality of the evaluation target data are arranged on a display device; a step of moving one of the markers selected from the plurality of markers in response to an instruction input by the evaluator by the information processing device; a step of the information processing device reproducing, in the reproduction device, one of the evaluation target data corresponding to one of the markers selected from the plurality of markers in response to an instruction input by the evaluator; The information processing device acquires evaluation results for the plurality of evaluation targets. having the step of displaying the marker placement area includes displaying the marker placement area including one or more index guide elements that guide a predetermined relationship between a position of the marker in the marker placement area and a value of the index for each of one or more indexes related to the evaluation of the evaluation object; The step of acquiring the evaluation result for one of the evaluation targets includes: acquiring positions of the markers in the marker placement area; Or, acquiring the positions of the plurality of markers in the marker arrangement area, and acquiring one or more values ​​of the index for each of the plurality of evaluation objects based on the acquired positions of the markers and the predetermined relationship; Including, method.

2. a step of changing a display mode of one of the markers selected from the plurality of markers to a display mode distinguishable from the other markers when the information processing device is reproducing the evaluation target data corresponding to the one of the markers selected from the plurality of markers; The method of claim 1.

3. the step of reproducing the evaluation target data includes reproducing one of the evaluation target data corresponding to one marker at least one of when one of the markers is selected in response to an instruction input by the evaluator and when the one marker is moved in response to an instruction input by the evaluator; The method of claim 2.

4. the step of reproducing the evaluation target data includes, when the evaluation target data corresponding to one of the markers selected from the plurality of markers is reproduced, when an instruction from the evaluator to deselect the one marker is input, stopping the reproduction of the evaluation target data; The method according to claim 3.

5. The information processing device includes a step of setting an operation mode related to the reproduction of the evaluation target data to a reproduction enabled mode in which the evaluation target data can be reproduced or a reproduction inhibiting mode in which the reproduction of the evaluation target data is inhibited in response to an instruction input by the evaluator, the step of reproducing the evaluation target data includes, when the reproduction inhibition mode is set, inhibiting reproduction of the evaluation target data regardless of an instruction input by the evaluator; The method according to claim 3.

6. the information processing device includes a step of setting the number of times the evaluation target data is to be repeatedly played back in response to an instruction input by the evaluator; the step of reproducing the evaluation target data includes repeatedly reproducing the one evaluation target data a set number of times each time an instruction to reproduce the one evaluation target data is input by the evaluator, The method according to claim 3.

7. the step of setting the number of repeated playbacks includes setting a loop playback in which the evaluation target data is repeatedly played back without limiting the number of repeated playbacks in response to an instruction input by the evaluator; the step of playing back the evaluation target data includes, when the loop playback is set, repeatedly playing back one piece of evaluation target data in response to an instruction from the evaluator to play back the one piece of evaluation target data, without limiting the number of times the evaluation target data is repeatedly played back; The method according to claim 6.

8. the information processing device includes a step of setting a playback order for two or more of the markers selected from a plurality of the markers in response to an instruction input by the evaluator; the step of playing back the evaluation target data includes, when the playback order is set for two or more of the markers, playing back the two or more evaluation target data in an order according to the playback order in response to an instruction from the evaluator to play back the two or more evaluation target data corresponding to the two or more markers; The method of claim 2.

9. the information processing device includes a step of setting a part of an entire playback section in one of the evaluation target data as a playback target section in response to an instruction input by the evaluator; the step of reproducing the evaluation target data includes, when the reproduction target section is set for one of the evaluation target data, reproducing the reproduction target section in the one of the evaluation target data in response to an instruction from the evaluator to reproduce the one of the evaluation target data; The method of claim 2.

10. The step of displaying the marker placement area includes: the index guide element providing guidance on the relationship between a value of one coordinate representing a position on a line on which the marker is placed and a value of one of the indices; the index guide element for providing guidance on the relationship between two coordinate values ​​representing a position on a plane on which the marker is placed and the values ​​of the two indices; the index guide element for guiding the relationship between three coordinate values ​​representing a spatial position where the marker is placed and the values ​​of the three indices; displaying the marker placement area including at least one of The method of claim 1.

11. a plurality of marker-related images in one-to-one correspondence with a plurality of said markers; A step in which the information processing device displays the marker-related image in the marker arrangement area, replacing each of the plurality of markers with a corresponding marker-related image in response to an instruction input by the evaluator; displaying the marker-related image corresponding to one of the markers selected from the plurality of markers in response to an instruction input by the evaluator; displaying a plurality of the markers and a plurality of the marker-related images in association with each other in a display manner in which one of the markers and one of the marker-related images in a corresponding relationship have a display manner in which they are related to each other and are distinguishable from the other markers and the other marker-related images in response to an instruction input by the evaluator; The method includes at least one of the steps of: The method of claim 1.

12. the information processing device sets a display state of each of the markers to a first display state or a second display state that is lighter than the first display state in response to an instruction input by the evaluator; the step of displaying the marker placement area includes displaying each of the markers in the first display state or the second display state in accordance with the setting of the display state. The method of claim 1.

13. the information processing device includes a step of assigning a plurality of different hierarchies to a plurality of the markers in response to an instruction input by the evaluator; a display order in the marker placement area is determined for the plurality of hierarchical layers; the step of displaying the marker placement area includes, when two or more of the markers are located at the same location in the marker placement area, displaying, at the location, the marker to which the hierarchical level with the highest display rank is assigned among the two or more markers; The method of claim 12.

14. the step of moving the marker includes, when two or more of the markers are located at a location specified by an instruction input by the evaluator, selecting the marker to which the hierarchical level with the highest display rank is assigned among the two or more markers, and moving the selected marker; the step of reproducing the evaluation target data includes, when two or more of the markers are located at a location specified by an instruction input by the evaluator, selecting the marker to which the hierarchical level with the highest display rank is assigned from among the two or more markers, and reproducing the evaluation target data for the selected marker; The method of claim 13.

15. The step of moving the marker includes, as evaluation process information relating to the process of evaluation by the evaluator, The number of movements of each of the markers; The distance traveled by each of the markers; and The movement time of each of the markers; Information regarding the trajectory of movement of each of the markers; obtaining at least one of The method of claim 1.

16. the step of reproducing the evaluation target data includes acquiring the number of reproductions of each of the markers as evaluation process information relating to the process of evaluation by the evaluator; The method of claim 1.

17. A program including instructions for causing an information processing device to perform a process for supporting an evaluator in providing a subjective evaluation to each of a plurality of evaluation targets, the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; The processing performed by the information processing device in accordance with the command includes each step of the method according to any one of claims 1 to 16. program.

18. An information processing device that supports an evaluator in giving a subjective evaluation to each of a plurality of evaluation targets, comprising: the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; A processing section; a memory unit that stores instructions to be executed by the processing unit; The processing performed by the processing unit in accordance with the instructions includes each step of the method according to any one of claims 1 to 16. Information processing device.

19. A system for supporting an evaluator in giving a subjective evaluation to each of a plurality of evaluation targets, comprising: the evaluation object is a perceptual object including at least one of a sound, a still image, and a video obtained by playing evaluation object data including at least one of a sound data, a still image data, and a video data on a playback device; One or more information processing devices; A server device, The information processing device has a means for performing each step of the method according to any one of claims 1 to 16, the server device collects the evaluation results obtained in the step of acquiring evaluation results for the plurality of evaluation targets from one or more of the information processing devices; system.

Images