Information processing system, information processing method, program, and information processing device

Abstract

An information processing system according to one embodiment of the present invention is provided with a first acquisition unit, a second acquisition unit, a status information generation unit, and a support processing unit. The first acquisition unit acquires a captured image in which a musical instrument is captured. The second acquisition unit acquires information on consultation items relating to the captured function of the musical instrument. The current situation information generation unit generates current situation information relating to the current situation of the musical instrument on the basis of the captured image, the current situation information including information relating to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument. The support processing unit generates and outputs support information for solving the consultation item on the basis of the information of the consultation item and the current situation information.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, a program, and an information processing apparatus applicable to a support system related to musical instruments. Background Art

[0002] Patent Documents 1 and 2 disclose systems that enable communication between a teacher and a student via a network and conduct lessons for the purpose of improving musical performance skills.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2004-252391

[0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2003-208089 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] Not only is there a demand for improvement in playing skills, but also for technology that can support the resolution of inquiries related to musical instruments.

[0009] In view of the above circumstances, one object of the present invention is to provide an information processing system, an information processing method, a program, and an information processing device that can efficiently support the resolution of consultation matters related to the functions of musical instruments.

[0010] Means for solving problems

[0011] In order to achieve the above-mentioned object, an information processing system according to one aspect of the present invention includes a first acquisition unit, a second acquisition unit, a presence information generation unit, and a support processing unit.

[0012] The first acquisition unit acquires an image of a musical instrument.

[0013] The second acquisition unit acquires information on consultation matters related to the function of the photographed musical instrument.

[0014] The current information generating unit generates current information related to the current status of the musical instrument based on the captured image, the current information including information related to at least one of a structure of the musical instrument, a state of the musical instrument, and an installation location of the musical instrument.

[0015] The support processing unit generates and outputs support information for solving the consultation matter based on the information on the consultation matter and the current situation information.

[0016] An information processing method according to one embodiment of the present invention is an information processing method executed by a computer system, and includes acquiring an image of a musical instrument.

[0017] Information on consultation matters related to the function of the photographed musical instrument is obtained.

[0018] Based on the captured image, current status information related to the current status of the musical instrument is generated, the current status information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument.

[0019] Based on the information about the consultation matter and the current situation information, support information for solving the consultation matter is generated and output.

[0020] A program involved in one embodiment of the present invention causes a computer system to perform the following operations: obtaining a photographic image of a musical instrument; obtaining information on consultation matters related to the function of the photographed musical instrument; generating current status information related to the current status of the musical instrument based on the photographic image, the current status information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation location of the musical instrument; and generating and outputting support information for solving the consultation matters based on the information on the consultation matters and the current status information.

[0021] An information processing device according to one aspect of the present invention includes the first acquisition unit, the second acquisition unit, the presence information generation unit, and the support processing unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram showing a configuration example of the musical instrument association support system according to the first embodiment.

[0023] Figure 2 This is a block diagram showing a functional configuration example of a server device.

[0024] Figure 3 This is a flowchart showing a basic operation example of the server device.

[0025] Figure 4 It is a diagram showing an example of support history information stored in the support history information DB.

[0026] Figure 5 It is a schematic diagram showing another example of support history information.

[0027] Figure 6 It is a diagram showing an example of FAQ information stored in the FAQ information DB.

[0028] Figure 7 This is a schematic diagram showing an example of a process image.

[0029] Figure 8 It is a schematic diagram showing an example of photographing guide information.

[0030] Figure 9 It is a schematic diagram showing another example of photographing guide information.

[0031] Figure 10 It is a schematic diagram showing a configuration example of a musical instrument association support system according to the second embodiment.

[0032] Figure 11 It is a schematic diagram showing a configuration example of a musical instrument association support system according to a third embodiment.

[0033] Figure 12 This is a block diagram showing a hardware configuration example of a computer that can be used as a consultation terminal, a support terminal, and a server device. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0035] <First embodiment>

[0036] [Musical Instrument Association Support System]

[0037] Figure 1 1 is a schematic diagram showing a configuration example of a musical instrument related support system according to a first embodiment of the present invention. The musical instrument related support system 1 can provide efficient support for solving various consultation matters related to musical instruments 2. Figure 1 In the illustrated example, an electronic drum is illustrated as the musical instrument 2 , but the type of the musical instrument 2 is not limited, and any musical instrument can be supported by the musical instrument association support system 1 .

[0038] For example, a player 3 playing a musical instrument 2 can receive support for solving various consultation matters by using the musical instrument-related support system 1. For example, with the musical instrument 2 at hand, the player 3 can easily receive high-precision support on the spot.

[0039] Furthermore, the person who can use the musical instrument support system 1 is not limited to the player 3 of the musical instrument 2. The player 3's family members, acquaintances, and other people who may have consultation matters related to the musical instrument 2 can also use the musical instrument support system 1.

[0040] Furthermore, the musical instrument support system 1 can also be used by support staff 4 of a musical instrument manufacturer, etc., who provides so-called user services to players 3 and others of musical instruments 2. By using the musical instrument support system 1, the support staff 4 can obtain information useful for support services, thereby providing high-quality support services to players 3 and others. Furthermore, the workload associated with support services can be reduced.

[0041] In this disclosure, the term "user" will be used to refer to anyone who can utilize the instrument-related support system 1, regardless of whether they are the person providing consultation or the person providing support. Furthermore, a user providing consultation will be referred to as a "consulting user." Furthermore, a user providing support in resolving the consultation will be referred to as a "supporting user."

[0042] The "consulting user" may include, for example, the player of the musical instrument 2, the owner of the musical instrument 2, their family members, acquaintances, a person who lives in the space where the musical instrument 2 is located, a person who lives near the space where the musical instrument 2 is located, a person to whom the handling of the musical instrument 2 has been entrusted, and any other person who may have a consultation matter. The "consulting user" may also be simply referred to as a consultant.

[0043] As a "support side user", for example, it can be equivalent to any person who may receive consultation related to the musical instrument 2, such as support staff 4 of the musical instrument manufacturer, on-site staff of the musical instrument manufacturer, music teachers in schools, teachers in music classrooms, clerks in musical instrument stores, etc.

[0044] These "consulting users" and "supporting users" can use the musical instrument support system 1 to solve various consulting matters. That is, the musical instrument support system 1 is a system that can exert beneficial effects on both "consulting users" and "supporting users".

[0045] Furthermore, there may be a situation where a relationship between a consulting party and a consulting party exists between players of the musical instrument 2 (e.g., senior and junior players). In such a case, the player receiving the consulting party can also serve as a "supporting user" and utilize the musical instrument association supporting system 1.

[0046] That is, the player of the musical instrument 2, the owner of the musical instrument 2, their family members, acquaintances, people who use the space where the musical instrument 2 is located as their living space, people who live near the space where the musical instrument 2 is located, people who are entrusted with the disposal of the musical instrument 2, etc. can also serve as "support side users" and utilize this musical instrument association support system 1.

[0047] In addition, the teacher of the music classroom may also want to consult about the musical instrument. In such a case, the teacher of the music classroom can also serve as a "consulting side user" and utilize this musical instrument association support system 1 to consult the support staff of the musical instrument manufacturer etc.

[0048] That is, any person who may receive consultation related to musical instrument 2, such as support staff of musical instrument manufacturers, on-site staff of musical instrument manufacturers, music teachers in schools, teachers in music classrooms, clerks in musical instrument stores, etc., can also use this musical instrument association support system 1 as a "consulting side user".

[0049] like Figure 1 As shown, in this embodiment, the musical instrument association support system 1 includes a consulting terminal 5 , a support terminal 6 , and a server device 7 .

[0050] The consulting terminal 5, the support terminal 6, and the server device 7 are communicatively connected via a network 8. The network 8 is constructed, for example, by the Internet or a wide area communication network. Alternatively, any WAN (Wide Area Network) or LAN (Local Area Network) may be used, and the protocol used to construct the network 8 is not limited.

[0051] The consulting terminal 5, the supporting terminal 6 and the server device 7 have hardware necessary for a computer, such as a processor such as a CPU, a GPU, or a DSP, a memory such as a ROM, a RAM, or a storage device such as a HDD (see Figure 12 The processor loads the program according to the present technology stored in the storage unit or the memory into the RAM and executes the program, thereby executing the information processing method according to the present technology (the method for supporting the resolution of consultation matters).

[0052] In addition, in this embodiment, a computer with a photographing function is used as the consulting terminal 5. Figure 1 In the example shown, a smartphone equipped with a camera (not shown) is used as the consulting terminal 5 .

[0053] As the camera, for example, a digital camera equipped with an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor can be used. In addition, any imaging device having any structure can be installed.

[0054] Furthermore, computers with a camera function are not limited to smartphones and can employ any configuration. For example, tablet computers, HMDs (head-mounted displays) such as AR (augmented reality) glasses or VR (virtual reality) glasses, and PCs can also be used.

[0055] Furthermore, in this embodiment, the smartphone serving as the consulting terminal 5 is equipped with a touch panel 9. Touch panel 9 functions as both a display unit and an input unit. Various image information, such as icons, text, a GUI (Graphical User Interface), and captured images, are displayed on touch panel 9. Furthermore, this embodiment also enables AR display, where AR objects (virtual images) are superimposed on real objects.

[0056] In addition, in this disclosure, an image includes both a still image and a moving image (video). In this embodiment, a moving image including audio information can be captured as a captured image by the camera of the consulting terminal 5 .

[0057] For example, Figure 1 As shown, the entire electronic drum, which is the musical instrument 2, can be photographed so that it is included in the field of view. Of course, the present invention is not limited to this, and a portion of the musical instrument 2 can be zoomed in and photographed, or the musical instrument 2 can be photographed from various shooting directions, such as from the front, from above, from the left and right, or from an angle.

[0058] Alternatively, the consulting terminal 5 can be fixed at a predetermined position to capture a performance image of the performer 3 playing the musical instrument 2. This performance image includes the sound of the musical instrument 2. Furthermore, the musical instrument 2 can be captured while also including the peripheral devices used with the musical instrument 2. Of course, it is also possible to zoom in on only the peripheral devices and capture the image.

[0059] Furthermore, the performer 3 can perform various touch operations and text input operations on the GUI and icons displayed on the touch panel 9. This allows the performer 3 to input a variety of information. Furthermore, the consulting terminal 5 can also be equipped with various devices such as a microphone, speaker, GPS, and IMU (Inertial Measurement Unit) sensors.

[0060] In addition, a computer equipped with a display 10 capable of displaying various image information is used as the support-side terminal 6. Figure 1In the example shown, a notebook PC is used as the support-side terminal 6. However, the present invention is not limited thereto, and any computer such as a smartphone, a tablet terminal, or an HMD (Head Mounted Display) such as AR glasses or VR glasses can be used.

[0061] Furthermore, in this embodiment, the consultation terminal 5 and the support terminal 6 each have a communication function. Furthermore, a communication function is enabled between the performer 3 and the support staff 4. The specific structure and method for enabling communication between the two devices are not limited, and any structure and method may be employed.

[0062] The server device 7 is configured to realize the present musical instrument association support system 1, and any computer such as a PC can be used. Figure 1 As shown, in this embodiment, a database (DB) 11 is constructed and accessible to the server device 7. The DB 11 functions as a storage unit, storing various information related to the musical instrument association support system 1. The DB 11 can be constructed in a storage device included in the server device 7 or in a storage device configured externally to the server device 7.

[0063] For example, the performer 3 downloads an application (application program) for using the musical instrument association support system 1 to the consulting terminal 5. For example, the performer 3 enters information such as an ID and password to create an account for using the musical instrument association support system 1. Of course, creating an account is not necessary.

[0064] The performer 3 starts up an application for utilizing the musical instrument association support system 1. Then, the performer 3 selects a musical instrument photographing mode and uses the photographing function of the consulting terminal 5 to photograph the musical instrument 2 that is the subject of the consultation.

[0065] Furthermore, performer 3 selects the inquiry input mode and, to enter inquiries related to the photographed musical instrument 2, uses the input screen. The specific method for entering inquiries is not limited, and any method such as text input, voice input, or questionnaire input (information input based on a questionnaire format) can be used. Furthermore, the method and GUI used by performer 3 to enter input information are not limited and can be designed arbitrarily.

[0066] Alternatively, for example, by turning on the call function within the application, a call can be established with the support staff 4. Furthermore, it is also possible to verbally communicate the consultation matter to the support staff 4. In this case, the support staff 4 inputs the information of the performer 3's consultation matter via the support-side terminal 6.

[0067] The captured image of the musical instrument 2 and the information on the consultation item are transmitted to the server device 7 via the network 8. The server device 7 then generates status information on the status of the musical instrument 2 and performs various support processes for solving the consultation item.

[0068] For example, images and texts supporting the solution of the consultation are output as support information to the touch panel 9 of the consultation terminal 5 used by the performer 3. The performer 3 can perform operations required for solving the consultation while checking the support information displayed on the touch panel 9.

[0069] Furthermore, the display 10 of the support terminal 6 used by the support staff 4 displays images, texts, etc. useful for supporting the performer 3 as support information. The support staff 4 can explain and instruct the performer 3 based on the support information displayed on the display 10 .

[0070] The support information output to the consulting terminal 5 and the supporting terminal 6 may utilize current status information related to the current status of the musical instrument 2. Furthermore, the same image, text, etc. may be output as the support information output to the consulting terminal 5 and the supporting terminal 6. Furthermore, audio information may be generated and output as audio as the support information.

[0071] In addition, we can provide flexible support for various consultation matters and efficiently support the resolution of consultation matters.

[0072] [Functional Configuration Example of Server Device]

[0073] Figure 2 7 is a block diagram showing a functional configuration example of the server device 7. Figure 2 As shown, in this embodiment, the first acquisition unit 13, the second acquisition unit 14, the presence information generation unit 15, the support processing unit 16, and the questionnaire unit 17 are configured as functional blocks by executing a predetermined program on the processor of the server device 7. Of course, dedicated hardware such as an IC (integrated circuit) may be used to implement the functional blocks.

[0074] The program is installed on the server device 7 via various recording media, for example. Alternatively, the program can be installed via the Internet, etc. The type of recording medium that records the program is not limited, and any computer-readable recording medium can be used. For example, any computer-readable non-transitory storage medium can be used.

[0075] The first acquisition unit 13 acquires a captured image of the musical instrument 2. In the present embodiment, the captured image of the electronic drum captured by the performer 3 using the consulting terminal 5 is transmitted to the server device 7 via the network 8.

[0076] In this embodiment, a moving image including sound information is captured as the captured image by the consulting terminal 5. Therefore, when the consulting terminal 5 captures a performance image, the first acquiring unit 13 can acquire information on the performance sound of the musical instrument 2.

[0077] Furthermore, performance sound information includes, for example, performance data obtained by recording the performance sounds of acoustic instruments, etc., using a microphone of the consulting terminal 5 or the like. However, the information is not limited to this. MIDI (registered trademark) (Musical Instrument Digital Interface) data acquired from performances of electronic instruments such as electronic drums, and wireless MIDI data using Bluetooth (registered trademark) or Wi-Fi (registered trademark) are also included in the performance sound information.

[0078] The second acquisition unit 14 acquires information regarding consultation items related to the photographed musical instrument 2. In this embodiment, information regarding consultation items related to the functions of the musical instrument 2 is primarily acquired. For example, information regarding consultation items such as worries and concerns regarding the inherent functions of the musical instrument 2 is acquired. Of course, this is not limited to this, and the musical instrument-related support system 1 can be applied to any consultation item regarding the musical instrument 2.

[0079] For example, information on consultation items input by the performer 3 into the consultation terminal 5 is transmitted to the server device 7 via the network 8 and acquired by the second acquisition unit 14. Alternatively, the consultation items may be verbally communicated to the support staff 4 via the call function. The communicated consultation items may also be input into the support terminal 6, transmitted to the server device 7 via the network 8, and acquired by the second acquisition unit 14.

[0080] Alternatively, information on the inquiry item can be obtained by analyzing the captured image acquired by the first acquisition unit 13. For example, the apparent damage to the appearance of the musical instrument 2 can be determined by analyzing the captured image. Furthermore, the damage to the musical instrument 2 can be inferred as information on the inquiry item.

[0081] Furthermore, it is possible to determine by analyzing the captured image (particularly a moving image) whether the player 3 has clearly made an error in the operation of the musical instrument 2. Furthermore, it is also possible to infer information related to the error in the operation as information on the inquiry matter.

[0082] That is, the second acquisition unit 14 can acquire information on consultation matters based on at least one of the input information input from the performer 3 and the captured image.

[0083] In this disclosure, acquiring information, etc., includes not only receiving the information from other devices but also generating the information by performing analysis, extraction, inference, etc. In other words, acquiring information on consultation matters includes both receiving information on consultation matters and generating information on consultation matters.

[0084] The current information generating unit 15 generates current information on the current status of the musical instrument based on the captured image of the musical instrument 2 . The current information includes information on at least one of the structure of the musical instrument 2 , the state of the musical instrument 2 , and the installation position of the musical instrument 2 .

[0085] The information related to the structure of the musical instrument 2 includes arbitrary information indicating the structure of the musical instrument 2. For example, the information related to the structure of the musical instrument 2 includes information such as the type of the musical instrument 2, the model of the musical instrument 2, the product number of the musical instrument 2, and the year of manufacture of the musical instrument 2.

[0086] The information related to the status of the musical instrument 2 includes any information indicating the status of the musical instrument 2. For example, the information related to the status of the musical instrument 2 includes information on the setting status of the musical instrument 2. Furthermore, if the musical instrument 2 is an electronic musical instrument or is used with an electronic device, information on the setting status of parameters such as volume and sound, and version information on the firmware of the electronic device, software such as an operating system, etc., is also included in the information related to the status of the musical instrument 2.

[0087] The information related to the installation location of the musical instrument 2 includes arbitrary information indicating the location or space where the musical instrument 2 is installed. For example, it includes information indicating whether the location where the musical instrument 2 is installed is indoors or outdoors. Furthermore, the information related to the installation location of the musical instrument 2 includes information such as whether the location where the musical instrument 2 is installed is a live house, a hall, or a stadium.

[0088] Furthermore, information regarding the structure of the musical instrument 2, information regarding the state of the musical instrument 2, and information regarding the installation location of the musical instrument 2 are not clearly defined. It is possible that one piece of information is included in both the information regarding the structure of the musical instrument 2 and the information regarding the state of the musical instrument 2. For example, in a musical instrument 2 constructed by assembling multiple parts, information regarding the assembled state of the multiple parts is included in both the information regarding the structure of the musical instrument 2 and the information regarding the state of the musical instrument 2.

[0089] For example, about Figure 1The electronic drum shown includes information related to the settings indicating how the pads for the treble tom, bass tom, floor tom, and cymbals are configured, in information related to the structure of the electronic drum, and also in information related to the status of the electronic drum (setting status).

[0090] Of course, arbitrary information related to the current status of the musical instrument 2, in addition to information related to the structure of the musical instrument 2, information related to the state of the musical instrument 2, and information related to the installation location of the musical instrument 2, may also be generated as the current status information. For example, information related to the current value of the musical instrument 2 (such as a second-hand transaction price), information on the current number of musical instruments 2, information on future production schedules, and other arbitrary information related to the current status of the musical instrument may be included in the current status information.

[0091] Furthermore, in this embodiment, current status information can be generated based on the musical performance information acquired by the first acquisition unit 13. For example, the musical performance information acquired by the first acquisition unit 13 can be used as current status information. Alternatively, the musical performance information can be processed by sound extraction, sound source separation, frequency analysis, and the like. The results of such processing can also be used as current status information.

[0092] The information on the consultation items may also be used to generate the current situation information. That is, the current situation information may be generated based on the information on the consultation items.

[0093] The support processing unit 16 executes various support processes for supporting the resolution of consultation matters. For example, based on the consultation matter information and the current situation information, it generates and outputs various support information for resolving the consultation matter.

[0094] The support information is useful for consulting users such as the performer 3 in resolving their consultations, and includes consulting support information output from the consulting terminal 5. Furthermore, the support information is useful for supporting users in performing support tasks, and includes supporting support information output from the supporting terminal 6.

[0095] Furthermore, there is no clear distinction between consulting-side support information and support-side support information. Support information that is useful to consulting-side users may also be useful to support-side users. In other words, support information that is both consulting-side support information and support-side support information may also be useful.

[0096] In this embodiment, a shared screen mode can be selected as an application of the musical instrument association support system 1. The performer 3 and the support staff 4 reach a consensus on the selection of the shared screen mode. In this manner, the touch panel 9 of the consulting terminal 5 and the display 10 of the support terminal 6 form a shared screen window. The images, pointers, and other items displayed in the shared screen window can be viewed and heard identically by both the performer 3 and the support staff 4.

[0097] For example, a captured image of the instrument 2 captured by the performer 3 is displayed in a shared screen window. Various support images for resolving inquiries can also be superimposed and displayed in conjunction with this image. For example, a process image indicating the process to be performed on the instrument 2, a correct answer image indicating the correct answer to the inquiry, and the like can be generated as virtual images (AR images) and superimposed on the captured image of the scene. Alternatively, a pointer image such as a cursor operated by the support staff 4 can be superimposed on the captured image of the scene. Such processing is also possible.

[0098] While checking the shared screen window, performer 3 can perform operations required to resolve inquiries, for example. Furthermore, they can ask questions to support staff 4 in real time. Support staff 4 can efficiently explain and instruct performer 3 while checking the shared screen window. Furthermore, the various support images displayed in the shared screen window constitute one embodiment of the support information involved in this technology.

[0099] For example, by performing AR display in which the correct state of the instrument 2 is superimposed on the current state of the instrument 2, the player 3 can easily determine whether the correct state is the same. In addition, it becomes easy to guide the current state of the instrument 2 to the correct state.

[0100] Furthermore, both parties can easily communicate using the call function while checking the shared screen window, such as "Please press this button," "Is it this button?" or "No, it's the button to the right of that button." Furthermore, they can easily issue a performance instruction such as "Please hit these three points: snare drum, kick drum, and hi-hat," and obtain information about the current performance sound.

[0101] Furthermore, as support information, instruction information for pressing a predetermined key, instruction information for instructing the performance of the musical instrument 2, etc. can be generated and output to the consulting terminal 5. Furthermore, an instruction for the performer 3 to press a predetermined key, or an instruction for the performer 3 to perform the musical instrument 2 can also be output to the support staff 4 as support information.

[0102] In addition, various support processes such as analysis, resolution, retrieval, and inference can be performed to support the resolution of various consultation matters. The results of these support processes and the information and data output to the consultation terminal 5 and the support terminal 6 are included in the support information involved in this technology.

[0103] The questionnaire unit 17 outputs questionnaires related to the consultation items and obtains information on responses to the questionnaires. Questionnaires related to the consultation items include a consultation item questionnaire for inputting the consultation items, a musical instrument status questionnaire for generating information on the current status of the musical instrument 2, or a support questionnaire for generating support information. Of course, these various questionnaires are not clearly defined. It is also possible to generate a questionnaire that can function as both a consultation item questionnaire and a musical instrument status questionnaire.

[0104] Typically, the questionnaire is presented to the consulting user. Therefore, the questionnaire is output to the consulting terminal 5. For example, regarding the specific content of the consultation matter, the consultation matter questionnaire is output in a manner that assists the consulting user in inputting information. Furthermore, the instrument status questionnaire is output in a manner that assists in generating status information, such as obtaining information related to the current status of the instrument 2, such as the structure, condition, and location of the instrument 2, which is the subject of the consultation matter, as answer information.

[0105] In addition, a support questionnaire is output in a manner such as obtaining the consulting user's knowledge related to the musical instrument 2, other musical instruments owned by the consulting user, information on peripheral devices, information on operations that the consulting user can perform, etc. as answer information, and assisting in the resolution of the consulting matter.

[0106] Of course, a questionnaire can also be provided to the user on the support side to obtain information about the musical instrument 2, other musical instruments that can be prepared, and peripheral equipment. The method of presenting various questionnaires is not limited, and any GUI can be configured. In addition, methods other than GUI display, such as voice input for voice announcements, can also be used.

[0107] By using the information provided in response to the questionnaire, it is possible to improve the accuracy of obtaining information on consultation matters, generating current status information, and generating and outputting support information, thereby increasing the efficiency and accuracy of support processing.

[0108] Any technology (algorithm, etc.) can be used to obtain information on consultation matters, generate current status information, generate support information, etc. For example, any machine learning algorithm using DNN (Deep Neural Network), RNN (Recurrent Neural Network), CNN (Convolutional Neural Network), etc. can also be utilized. For example, by using AI (artificial intelligence) that performs deep learning, each process can be performed with high accuracy.

[0109] For example, by inputting a photographic image of a musical instrument 2, etc., into a learning model that has already undergone machine learning to output information about the consultation item, information about the consultation item can be obtained from the learning model. Furthermore, by inputting a photographic image of a musical instrument 2, etc., into a learning model that has already undergone machine learning to generate current status information, current status information can be obtained from the learning model. Furthermore, by inputting information about the consultation item and current status information, etc., into a learning model that has already undergone machine learning to output support information, support information can be obtained from the learning model.

[0110] As a learning method for learning models, for example, the error backpropagation method is used. This method is widely used for learning neural networks. Neural networks are models that mimic the neural circuits of the human brain and have a layered structure consisting of three layers: an input layer, an intermediate layer (hidden layer), and an output layer.

[0111] Neural networks with multiple intermediate layers are particularly known as deep neural networks. Deep learning techniques, used to train these networks, are known for their ability to learn complex patterns hidden in large amounts of data. Error backpropagation is one such learning method, often used in the training of CNNs, for example, used in image and video recognition.

[0112] Furthermore, as a hardware structure for realizing such machine learning, a neurochip or a neuromorphic chip in which the concept of a neural network is embedded can be used.

[0113] The algorithm used to train the learning model is not limited, and any machine learning algorithm can be used. For example, machine learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, inverse reinforcement learning, active learning, and transfer learning. In addition, machine learning models such as HMM (Hidden Markov Model) and SVM (Support Vector Machine) can also be used.

[0114] The learning block (learning unit) for learning the learning model may also be configured in the consulting terminal 5 or the support terminal 6. Alternatively, the learning block (learning unit) may be configured in a computer separate from the consulting terminal 5 or the support terminal 6, and the learned model, after training, may be installed in the consulting terminal 5 or the support terminal 6. The specific configurations of the learning model and the learning block (learning unit) for learning the learning model are not limited.

[0115] Furthermore, the application of machine learning algorithms can be performed on any process within the present disclosure. In other words, any process described in the present disclosure can be performed using machine learning.

[0116] Furthermore, the method for extracting various types of information, such as current status information and information on consultation matters, from the captured image of the musical instrument 2 is not limited, and any technique (algorithm) can be used. For example, any image recognition technique, such as matching processing using a model image of the object, edge detection, or projective transformation, can be utilized. Skeleton estimation (bone estimation) and the like can also be used. Furthermore, externally constructed libraries with existing image processing, machine learning, and other functions can also be utilized.

[0117] Any machine learning algorithm can be used to extract information from captured images. For example, by performing semantic segmentation on image information, it is possible to determine the type of object for each pixel in the image.

[0118] Figure 3This is a flowchart showing a basic operation example of the server device 7. The first acquisition unit 13 acquires an image of the musical instrument 2. Furthermore, the second acquisition unit 14 acquires information on consultation items related to the function of the photographed musical instrument 2 (step 101).

[0119] The current information generating unit 15 generates current information on the current state of the musical instrument 2 based on the captured image. The current information includes information on at least one of the structure, state, and installation position of the musical instrument 2 (step 102 ).

[0120] The support processing unit 16 generates and outputs support information for solving the consultation matter based on the consultation matter information and the current situation information (step 103 ).

[0121] [Example of changes in consultation matters]

[0122] Examples of the information on inquiry matters related to the function of the musical instrument 2 include inquiry matters related to the structure of the musical instrument 2 , inquiry matters related to the status of the musical instrument 2 , and inquiry matters related to the installation position of the musical instrument 2 .

[0123] Specific examples of these consultation matters include the following:

[0124] Inquiries regarding the structure of Musical Instrument 2 Don’t know how to assemble it, etc.

[0125] Inquiries regarding the status of Musical Instrument 2 Not sure whether the settings are correct, want to know the appropriate parameter settings, want to know the appropriate software version, etc.

[0126] Inquiries regarding the installation location of Musical Instrument 2 I hope to know where is the best setting.

[0127] In addition, information on consultation matters related to the functions of musical instruments may include, for example, consultation matters related to the pronunciation of musical instruments, consultation matters related to the movements of musical instruments when playing, consultation matters related to the setting of musical instruments, consultation matters related to malfunctions of musical instruments, consultation matters related to the maintenance of musical instruments, and consultation matters related to the extended functions of musical instruments.

[0128] Specific examples of these consultation matters include the following:

[0129] Inquiries regarding the pronunciation of musical instruments Unable to make a sound, unable to make the desired sound, making strange sounds, etc.

[0130] Inquiries regarding the movements of playing musical instruments A part of the instrument moves differently than intended, a part of the instrument does not move, a part of the instrument shakes, etc.

[0131] Inquiries regarding the setup of musical instruments Setting up is not smooth, not knowing how to set the parameters of electronic instruments, etc.

[0132] Inquiries regarding musical instrument malfunctions There is no sound (is it a malfunction?), and you want to fix the malfunction, etc.

[0133] Inquiries regarding maintenance of musical instruments Don't know the maintenance method, want to know better maintenance method, etc.

[0134] Inquiries regarding expanded functions of musical instruments You want to add instrument components (such as pads) to your electronic drum set, upgrade to higher-end accessories, or expand the functionality of your electronic instrument.

[0135] As described above, the musical instrument association support system 1 can efficiently support the resolution of various consultation matters.

[0136] Furthermore, the various types of information that constitute specific examples of the aforementioned consultation items are not clearly defined. A consultation item may also be about the structure of the musical instrument 2 and also about the condition of the musical instrument 2. Furthermore, a consultation item may also be about the sound of the musical instrument and also about a malfunction of the musical instrument.

[0137] [Performance-based support]

[0138] In this embodiment, the first acquisition unit 13 of the server device 7 can acquire information about the musical instrument 2's musical performance sound. The current information generation unit 15 can then generate current information based on the acquired musical performance sound. Furthermore, the support processing unit 16 can also generate support information based on the acquired musical performance sound.

[0139] For example, when the above-mentioned consultation item regarding the sound of a musical instrument is input, the generation of current information based on the information of the musical performance sound and the generation of support information based on the information of the musical performance sound are very effective.

[0140] [Support processing using support history information / FAQ information]

[0141] Figure 4This diagram illustrates an example of support history information 20 stored in support history information DB 19 constructed within DB 11. In this embodiment, various information, such as images and sounds, sent and received to resolve various consultation matters are recorded and stored through video or audio recording. For example, various support images displayed in a shared screen window, audio information, and conversation content are stored as history information.

[0142] like Figure 4 As shown, in this embodiment, support history information 20 includes the instrument ID, captured images, information on consultation matters, current status information, and support information. It may also include information such as the content of the call. This information (image data, audio data) can be stored directly, or it can be stored as support history information 20 with links to the information (image data, audio data) stored in other storage devices.

[0143] The musical instrument ID is an ID assigned according to the type of musical instrument. For example, an ID that can identify the type of musical instrument 2 such as acoustic drums, electronic drums, acoustic guitar, electric guitar, or piano is assigned.

[0144] For example, by analyzing a captured image of the musical instrument 2, the type of musical instrument 2 that is the subject of the consultation is determined, and an instrument ID corresponding to the type of musical instrument 2 is assigned. Alternatively, a user of the musical instrument association support system 1, such as a performer 3, may input information regarding the type of musical instrument to be consulted, and the instrument ID may be assigned based on the input information.

[0145] The captured image is a captured image of the selected instrument 2. For a single instrument 2, multiple captured images with different captured locations, shooting directions, etc. can be stored. Of course, a portion of the captured images can also be selected and stored.

[0146] The presence information is the presence information generated by the presence information generating unit 15. The support information is various types of support information generated and output by the support processing unit 16. The call content is the content of the call between the performer 3 and the support staff 4 via the call function.

[0147] In addition, information input by the performer 3, information on the questionnaire generated by the questionnaire unit 17, and information on responses to the questionnaire may be stored as the support history information 20. Furthermore, various information (text messages, call messages, etc.) exchanged between the performer 3 and the support staff may be stored as the support history information 20.

[0148] The support history information 20 is information including information on consultation matters received in the past and support information generated and output in the past to resolve the consultation matters.

[0149] exist Figure 4 In the example shown, the support history information 20 is managed for each instrument ID. For example, consultation matters related to electronic drums input by various users in the past are assigned instrument IDs corresponding to the electronic drums and managed as the support history information 20.

[0150] Thus, based on the musical instrument 2 that is the subject of the consultation, past support history information can be easily retrieved, and efficient support can be performed. In addition, for each musical instrument 2, analysis of what kind of consultation matters there are can be easily performed.

[0151] Figure 5 is a schematic diagram showing another example of the support history information 20. Figure 5 In the illustrated example, the support history information 20 is managed for each user ID assigned to a consulting-side user who seeks a solution to a consulting matter.

[0152] For example, a user ID is assigned to each consulting user when creating an account for utilizing the present musical instrument association support system 1. Then, consulting matters consulted by the same consulting user are managed based on the user ID.

[0153] This makes it easier to analyze what inquiries each consulting user has made in the past, enabling more efficient support. For example, it also makes it easier to determine whether the same inquiries have been made in the past, making support more efficient. Furthermore, when managing user information such as the region of residence, it also makes it easier to analyze which inquiries are most common for which musical instruments in each region.

[0154] Furthermore, the support history information 20 may be managed based on the support user ID. This facilitates analysis of how each support user supported solutions to inquiries in the past, and can be effectively utilized in establishing a future support business system.

[0155] Figure 6 This is a schematic diagram showing an example of FAQ information 23 stored in FAQ information DB 22 constructed within DB 11. FAQ information 23 includes pre-created consultation information and support information, and is a pair of consultation questions assumed to be common and support information for solving the consultation questions.

[0156] like Figure 6 As shown, the FAQ information 23 is managed based on the instrument ID corresponding to the type of the instrument 2, and stores consultation items and support information. Figure 4 as well as Figure 5 The support history information 20 shown is analyzed to select and generate information on frequently consulted matters. Proper management of the support history information 20 facilitates the generation of the FAQ information 23. Of course, the FAQ information 23 is not limited to being generated based on the support history information 20.

[0157] The FAQ information 23 is typically generated by processing information based on frequently asked questions without including personal information of the inquirer. The FAQ information 23 may be configured to be disclosed only to past support recipients, separate from information disclosed to all inquirers.

[0158] exist Figure 4 as well as Figure 5 The support history information 20 shown stores support information corresponding to the consulting user's solution actions. For example, when a consulting user attempts to resolve a consultation through trial and error, detailed support information is often stored to guide this trial and error until a solution is achieved. While there is a tendency for each consulting user to have a high degree of individuality, there are also many cases where very helpful support information is stored.

[0159] On the other hand, Figure 6 The FAQ information 23 shown tends to increasingly store general (universal) support information that guides the resolution of inquiries. In other words, there are also many cases where the information simply outputs the process leading to a resolution. Of course, such simple support information is useful if the inquiring user can fully understand it.

[0160] In the musical instrument related support system 1 , upon acquisition of consultation item information by the second acquisition unit 14 of the server device 7 , the support processing unit 16 can search the support history information 20 stored in the support history information DB 19 and output the search result as support processing.

[0161] For example, support history information 20 corresponding to the inquiry question is retrieved based on the captured image of the musical instrument 2, information on the inquiry question, and current situation information. Specifically, support history information 20 of the inquiry question having the same or similar content as the inquiry question is output as the search result.

[0162] As the output of the search result, all of the various information stored as the support history information 20 may be output. However, the present invention is not limited thereto, and only a part of the various information stored as the support history information 20, such as the support information, may be output.

[0163] For example, the consulting user can easily check support information from other people's past inquiries, etc. If other people have reached a solution through trial and error, they can check the detailed support information leading up to the solution, and can efficiently resolve the inquiry.

[0164] Furthermore, the supporting user can easily check the support information used to guide the problem to a solution in the past, and can use the support information as it is or further improve the support information to support the consulting user.

[0165] Furthermore, upon acquisition of the consultation item information by the second acquisition unit 14 of the server device 7 , the support processing unit 16 can search for the FAQ information 23 stored in the FAQ information DB 22 and output the search result as a support process.

[0166] For example, FAQ information 23 corresponding to the inquiry question is retrieved based on the captured image of the musical instrument 2, the inquiry question information, and the current situation information. Specifically, FAQ information 23 of the inquiry question with the same or similar content as the inquiry question is output as the search result.

[0167] As the output of the search results, all of the various information stored as the FAQ information 23 may be output. However, the present invention is not limited thereto, and only a part of the various information stored as the FAQ information 23, such as support information, may be output.

[0168] For example, the consulting user can easily check the FAQ information 23 corresponding to his / her own consulting matter. As a result, the consulting matter can be solved efficiently.

[0169] Furthermore, the supporting user can easily check the supporting information of the FAQ information corresponding to the inquiry from the consulting user and can use the supporting information as it is or further improve the supporting information to utilize it for supporting the consulting user.

[0170] Since both the consulting user and the support user do not need to search for appropriate support history information 20 and FAQ information 23 themselves, a highly efficient support system can be constructed.

[0171] Furthermore, the support information in the support history information 20 related to one's own consultation can be reviewed or reconfirmed at an appropriate time. This allows efficient resolution of the same consultation, preventing multiple consultations. Furthermore, skill-up in the instrument 2 can be achieved.

[0172] Furthermore, by providing a search function for the support history information 20 and the FAQ information 23 , it is possible to shorten the time, reduce the man-hours, and reduce the burden of support work for support users. Furthermore, it is possible to improve the quality of user support services.

[0173] Furthermore, the fact that the support history information 20 is accumulated over time also contributes to the efficiency of support for solving consultation matters.

[0174] Of course, the consulting user and the support user can each perform a search operation for the support history information 20 and the FAQ information 23. For example, the support processing unit 16 may perform a search for the support history information 20 and the FAQ information 23 based on the input of a search instruction by the consulting user and the support user.

[0175] [Support image generation]

[0176] The support processing unit 16 of the server device 7 can generate and output support images for solving consultation matters. For example, various images such as icons, GUIs, panel images, pointer images such as cursors, text images, and instruction manual images can be displayed.

[0177] Furthermore, the support processing unit 16 can superimpose the support image on the captured image of the musical instrument 2 or the model image of the musical instrument 2 and output the superimposed image.

[0178] For example, a virtual image (AR image) can be generated as a support image and superimposed on the actual instrument 2 included in the captured image. Furthermore, an image of the instrument 2 illustrated in its instruction manual, a photograph posted on a designated website, or an image using 3D model data of the instrument 2 can be displayed as a model image and superimposed on that image.

[0179] Furthermore, as a support image, a process image for indicating a process to be performed on the musical instrument 2 can be displayed. Any form can be adopted as the process image, for example, a form in which text images representing numbers indicating the process are displayed in sequence, a form in which pointer images are displayed in sequence at positions to be operated, a form including a text image explaining the process, and the like.

[0180] Figure 7 is a schematic diagram showing an example of a process image. Figure 7 In the example shown, an inquiry regarding a tuning method for an acoustic drum snare drum 25 is obtained. Then, an image of the striking surface of the snare drum 25 captured from the front is transmitted as a captured image.

[0181] The support processing unit 16 displays a digital virtual image 26 as a progress image, superimposed on the actual snare drum 25 and the positions of ten tuning bolts. Furthermore, a text image explaining the tuning method, such as "Please tighten the tuning bolts 1 / 4 turn at a time in the order shown on the left," is displayed as the progress image.

[0182] Instead of superimposing the virtual image of the actual snare drum 25 , the digital virtual image 26 may be superimposed on the model image of the snare drum 25 .

[0183] For example, the snare drum 25 is continuously filmed and displayed in shared screen windows on the consulting terminal 5 and the support terminal 6. In this state, a digital virtual image 26 can also be displayed as a progress image within the real-time video. Of course, the support staff 4 can also use the call function to provide support such as "Please tune in numerical order."

[0184] For example, while alternately viewing the digital virtual image 26 and the real snare drum 25 displayed in the shared screen window, performer 3 sequentially tightens the tuning bolts to tune the instrument. This process is displayed in the shared screen window. Support staff 4 can confirm performer 3's tuning progress in real time while providing support, such as "Is that correct?", "Please tighten the tuning bolt next to it slightly?", or "You're overtightening!" This allows for efficient and precise support.

[0185] Alternatively, the support staff 4 may be inexperienced and have little knowledge of tuning the snare drum 25, or may have forgotten how to tune the snare drum 25. In such a case, the support staff 4 can efficiently support the tuning of the snare drum 25 by checking the process image (digital virtual image 26) displayed in the shared screen window of the support-side terminal 6.

[0186] [Guide function for shooting of musical instruments]

[0187] In the present musical instrument association support system 1, a consulting user seeking a solution to a consultation issue photographs the musical instrument 2 that is the subject of the consultation and transmits the photographed image. Regarding the photographing of the musical instrument 2, the support processing unit 16 may output photographing guidance information for guiding the photographing of the musical instrument 2. The photographing guidance information may include, for example, information indicating the photographed portion of the musical instrument 2, the number of photographs, and the photographing direction. Furthermore, if the consulting terminal 5 is equipped with multiple cameras (telephoto / wide-angle / selfie, etc.), information indicating the appropriate camera may be output as photographing guidance information.

[0188] The specific form of the shooting guidance information is not limited, as long as it can guide the shooting location, number of shots, shooting direction, etc. of the musical instrument 2. For example, text images or audio messages such as "Please shoot the entire drum set from the front," "Please shoot the sound source module of the electronic drum from the front," or "Please shoot from the front, top, and back in this order" can be output as shooting guidance information. Alternatively, a virtual image of the part to be photographed, etc., can be output as shooting guidance information.

[0189] Figure 8 is a schematic diagram showing an example of photographing guide information. Figure 8 In the example shown, a virtual image 29 representing a sound source module of an electronic drum is displayed as shooting guidance information on a camera screen 28 in which instrument shooting mode is selected. Virtual image 29 is displayed superimposed on an actual object 30 (including the actual object 30 within the field of view) such as the sound source module being photographed. Furthermore, a text image or voice message such as "Please photograph the customer's sound source module by matching it to the sound source module displayed in the center of the screen" can also be output as shooting guidance information.

[0190] Figure 9 is a schematic diagram showing another example of shooting guide information. Figure 9 In the example shown, the display area of ​​the touch panel 9 is divided into left and right sections, forming a camera screen 31 for taking pictures and a sample screen 32. The camera screen 31 displays an actual object 33 within the field of view. The sample screen 32 displays a sample image 34 showing an example of a picture taken to support the resolution of the consultation. In addition, the sample screen 32 displays a text image (in the middle) that reads, "Please take a picture of the customer's sound source module as shown in the picture." Figure 9 In this way, sample images 34 and text images may be displayed as shooting guidance information. Of course, audio information may also be used.

[0191] The performer 3 can use the imaging guidance information to efficiently capture images that are effective in supporting the resolution of the consultation matter, thereby achieving efficient support.

[0192] The support processing unit 16 can also automatically set optimal shooting conditions. For example, it can automatically set optimal conditions for the part of the instrument 2 to be photographed, the number of images to be taken, and the shooting direction. Furthermore, if the consultation terminal 5 is equipped with multiple cameras (telephoto, wide-angle, Selfie, etc.), the optimal camera can be automatically selected. This allows the performer 3 to efficiently capture images that are effective in supporting the resolution of the consultation matter. The result is highly efficient support.

[0193] [Output of link information]

[0194] The support processing unit 16 can also output link information (e.g., URL (Uniform Resource Locator)) to a web page (website) used to resolve the inquiry as support information. Furthermore, it can also output link information (e.g., telephone number, link information to a web page) to facilities and stores that are useful for resolving the inquiry.

[0195] [Specific application example]

[0196] A specific application example of support for solving consultation matters using the musical instrument association support system 1 will be described.

[0197] (Application example 1: Inquiries regarding the pronunciation of musical instruments)

[0198] Content of inquiry: The sound of my electric guitar is not as good as that of a CD. It is loose and weak compared to the ideal CD sound.

[0199] Valid captured images include: images of the electric guitar being played, the amplifier connected to the electric guitar, images of peripheral devices such as the effects pedal connected to the electric guitar, etc. These images contain sound information, including the sound of the electric guitar being played.

[0200] Input information input by the performer 3: information of the desired music piece (CD title, song name, link information, characteristic information of the music piece, etc.).

[0201] The current status information generated by the current status information generating unit 15 includes: information related to the structure of the electric guitar, such as the type, model, and product number of the electric guitar; information related to the status of the electric guitar, such as the scale of the control knobs (paddles) of the electric guitar and the presence or absence of a capo; information related to the structure of peripheral devices, such as amplifiers and effects pedals, such as the type and product number of the peripheral devices; and information related to the status of the peripheral devices, such as the scale of the control knobs (paddles) of the amplifiers and effects pedals. In this disclosure, the current status information related to the peripheral devices is assumed to be included in the current status information of the musical instrument.

[0202] Furthermore, information on the musical sound of the electric guitar is acquired as current status information.

[0203] The support processing performed by the support processing unit 16 can compare the current electric guitar playing sound with the ideal sound. For example, frequency analysis can be performed on the sound information of both instruments and the analysis results compared. Additionally, any information useful for bringing the current electric guitar playing sound closer to the ideal sound can be detected. These comparison and analysis results are included in the support information.

[0204] Furthermore, the performance sound obtained by playing the musical instrument 2 may include sounds other than those intentionally produced by the performer 3. For example, in an acoustic instrument (such as an acoustic guitar), the sound produced by plucking the strings may include not only the sound produced by plucking the strings but also the sound of fingers rubbing against the strings when moving the strings, such as during chord changes. Furthermore, the sound may include noises caused by fingers accidentally touching the instrument body.

[0205] In addition, for example, when wearing headphones while playing an electronic musical instrument (such as an electronic keyboard instrument), there may be a situation where the performance sound heard from the headphones and the mechanical sound of the keyboard key operation heard by people other than the performer 3 are both included.

[0206] Furthermore, when playing a guitar or other instrument, the performer 3 may stomp their feet to keep the rhythm, and the stomping sound may be included in the performance sound. In addition, when using an amplifier or speaker as a peripheral device, the vibration sound (crackling) of the amplifier or speaker may be included in the performance sound.

[0207] Thus, the performance sound includes the actual performance sound of the instrument 2 intentionally produced by the performer 3 and other sounds produced along with the performance. Other sounds produced along with the performance include the action sounds of the performer 3 (sounds caused by the performance movements of the performer 3) and the action sounds of the instrument 2 (sounds caused by the movement of the instrument 2 during the performance).

[0208] For example, the sound produced by plucking the strings of an acoustic guitar or the like, and the sound heard through headphones on an electronic keyboard or the like are included in the true performance sound. The sound of the strings of an acoustic guitar or the like rubbing against the fingers, the noise caused by finger manipulation errors, and the stomping of the foot by the performer 3 are included in the action sound of the performer 3. The sound of operating the keys of an electronic keyboard or the like, and the vibration (crackling) sound of a speaker or the like are included in the action sound of the instrument 2.

[0209] In this musical instrument association support system 1, various sounds included in performance sounds can be appropriately extracted and separated for use in support processing. For example, the current status information generating unit 15 can extract and separate the actual performance sound, the movement sounds of the performer 3, and the movement sounds of the musical instrument 2 from the performance sound information acquired by the first acquisition unit 13, and generate the resulting state information as current status information. Alternatively, extraction and separation processing can be performed on sounds presumed to be effective (important) in resolving the consultation matter.

[0210] Furthermore, the support processing unit 16 may extract and separate the actual performance sound, the movement sound of the performer 3, and the movement sound of the musical instrument 2 from the performance sound information. Of course, the support processing unit 16 may also extract and separate sounds that are estimated to be effective (important) in solving the consultation matter.

[0211] For example, in this embodiment (Application Example 1), the actual performance sound is extracted and separated as the sound presumed to be effective (important) in resolving the consultation. This is then used by the support processing unit 16 for support processing such as the aforementioned comparison process. This enables even more accurate support.

[0212] In this manner, by having the current information generation unit 15 and the support processing unit 16 execute processing to extract and separate various sounds included in the performance sound, highly accurate support can be achieved.

[0213] Furthermore, as support processing, it is possible to detect information about the equipment needed to create a sound close to the ideal sound. For example, information about strings, picks, amplifiers, and effects pedals that are useful for achieving the desired sound can be generated as support information. Furthermore, it is possible to generate support information based on the use of existing amplifiers and effects pedals. For example, information about the settings of amplifier and effects pedal controls that are useful for achieving the desired sound can be generated and output as support information.

[0214] Furthermore, support information can be generated and outputted by linking to a know-how website (webpage) that provides detailed information on creating electric guitar sounds. Furthermore, support information can be generated and outputted by linking to a webpage for purchasing useful peripheral devices.

[0215] Furthermore, a process image for achieving ideal sound production can be generated and output as support information. For example, a process image can be generated and output as support information, showing how the control knobs of an electric guitar, an amplifier, and an effects pedal, as reflected in the captured image of the performer 3, are adjusted to their ideal settings. For example, a virtual image can be generated as the process image and superimposed on the captured image.

[0216] Of course, various support processes described above can be executed for the consultation matters, such as searching the support history information 20 and searching the FAQ information 23. This also applies to the application examples described below.

[0217] (Application Example 2: Inquiries regarding the pronunciation of musical instruments)

[0218] Content of inquiry: The sound of the recorder is abnormal. In particular, the "Fa" sound is not produced well.

[0219] Valid images include: Recorder footage. Recorder footage being played (including information about the sound being played). Specifically, footage of the recorder being played (the "fa" sound). Recorder footage with hands pressed (not being played). Specifically, footage showing how the hands are pressed to produce the "fa" sound.

[0220] The current status information generated by the current status information generating unit 15 includes: information related to the recorder's structure, such as the recorder model name and whether it is a German recorder or a Baroque recorder; information related to the recorder's status, such as the recorder's assembly state; and information on the recorder's played sound (particularly information on the "fa" sound).

[0221] The support processing performed by the support processing unit 16 can compare the pressing method (fingering) included in the captured image with the correct pressing method (fingering) for the German and Baroque styles. For example, image analysis technology can be used to determine whether the correct pressing method corresponding to the type of recorder played by the player 3 was used. The result of this determination is included in the support information.

[0222] Furthermore, as a supporting process, it is possible to determine whether the recorder is correctly assembled. For example, image analysis technology can be used to compare the recorder with the correct assembly state, allowing determination of correct assembly and any assembly anomalies. This determination result is included in the supporting information. Furthermore, if the assembly state is incorrect, gaps between the head, middle, and foot tubes can alter the pitch of the sound produced. Therefore, determining the assembly state is extremely effective.

[0223] Furthermore, as support information, information indicating whether the recorder played by the player 3 is German style or Baroque style can be generated and output. Furthermore, information indicating the correct pressing method (fingering) corresponding to each type can also be generated and output as support information.

[0224] Furthermore, as supporting information, a link to a technical knowledge website (webpage) that provides detailed information on pressing methods (fingering) for each type of recorder can be generated and output as supporting information. Furthermore, a link to a webpage for purchasing recorder instruction books can also be generated and output as supporting information.

[0225] Furthermore, as support processing, it is also possible to perform a determination as to whether the recorder is malfunctioning. The determination result is included in the support information. If the recorder is malfunctioning, information conveying this fact can also be generated and output as support information.

[0226] (Application Example 3: Inquiries regarding the pronunciation of musical instruments)

[0227] Content of inquiry: The electronic drum is silent (does not produce any sound).

[0228] Valid images include: Videos of electronic drums being played (including MIDI data), Videos of the sound generator module in action (including MIDI data), Videos showing the connection status of various cables (including connections to sockets, etc.).

[0229] The status information generated by the status information generating unit 15 includes: information related to the electronic drum's structure, such as the type, model, and product number of the electronic drum; information related to the electronic drum's status, such as the settings of the electronic drum pads and sound source module; and information related to the electronic drum's status, such as the equalizer and scale of the control knobs (paddles) of the sound source module.

[0230] The support processing performed by the support processing unit 16 can determine whether there is any abnormality in the cable connection state, notify the possibility that the connection cable is disconnected, determine whether the volume is set correctly, and determine whether there is an unintended connection in the headphone jack.

[0231] Furthermore, as support information, it is also possible to generate and output information prompting confirmation of whether the connection is correct. For example, prompting confirmation messages such as "Is the socket connected?", "Is the cable properly connected?", "Is the volume set as expected?", and "Is there something plugged into the headphone jack?" can be generated and output.

[0232] Of course, based on the above determination results, information indicating the correct setting, warning information, etc. can also be output. For example, messages such as "The socket is not connected, please connect it," "The volume is very low," or "Please unplug the headphones from the headphone jack" can also be generated and output as support information.

[0233] Furthermore, as support processing, support images can be generated and outputted, indicating cables, etc., that are likely to be the cause of a problem. Furthermore, support images showing the correct connection status can be generated and outputted. Furthermore, process images showing the process of setting settings such as volume can be generated and outputted. These support images (process images) can also be generated as virtual images and superimposed on the actual object in the captured image.

[0234] (Application example 4: Inquiries regarding the expansion of musical instrument functions)

[0235] I'm unsure if I can add a backing plate to my electronic drum set. I'm also unsure which models have backing plates guaranteed to function properly.

[0236] Valid images include: an overall image of the electronic drum, an image of the sound generator module in action (including MIDI data), and an image showing the connection status of various cables (including connections to sockets, etc.).

[0237] Input information input by the performer 3: Information about the pad to be added (model name, etc.) If an image of the pad to be added can be obtained, it can also be input as image information.

[0238] The current status information generated by the current status information generating unit 15 includes: information related to the electronic drum's structure, such as the type, model, and product number of the electronic drum; and information related to the sound source module's structure, such as the type, model, and product number of the sound source module. This information related to the sound source module's structure is included in the information related to the musical instrument's structure.

[0239] The support processing performed by the support processing unit 16 can determine whether a pad can be added. Furthermore, it can detect and output information about pads that have completed verification of sound source module operation. For example, it can query a database storing information about pads that have completed verification of sound source module operation, extract information related to whether the pad to be added can be used, restrictions on its use, and output this information as support information.

[0240] Furthermore, as support processing, information (e.g., a list of available underlays) that can be added to the sound source module in use can be presented. Furthermore, as support information, a link to a webpage associating the sound source module with the available underlays, or a link to a webpage for purchasing the available underlays can be generated and output.

[0241] (Application Example 5: Inquiries regarding the pronunciation of musical instruments)

[0242] Content of inquiry: I am experiencing unusual sounds from my electronic keyboard. The sound when I press the keyboard is unusual.

[0243] Valid captured images: Images of a performance (including the sound of the performance). Images of keyboard movements (including the sound of the performance). Images of the electronic keyboard instrument's movements (including the sound of the performance) when the tone is changed.

[0244] The current information generated by the current information generating unit 15 includes: information on the structure of the electronic keyboard instrument, such as the type, model, and product number of the electronic keyboard instrument; information on the musical sound of the electronic keyboard instrument; and information on the musical sound when the tone is changed.

[0245] Furthermore, the current status information generating unit 15 can also output instructions related to timbre change operations. In response to the timbre change instructions output via the consulting terminal 5, the performer 3 operates the control knobs (pick buttons) of the electronic keyboard instrument to change the timbre, while also capturing an image of the electronic keyboard instrument. Instructions for timbre change, etc., can also be executed by the support processing unit 16.

[0246] The support processing performed by the support processing unit 16 can compare the sound produced after a timbre change operation with the original sound information assumed by the change operation. It can also infer the cause of any abnormality in the sound produced after the change operation. For example, it can infer, as cause information, whether the timbre abnormality is caused by a malfunction or a parameter setting error. This inferred cause information is included in the support information. This inference of the cause makes it possible to distinguish between the electronic keyboard instrument's specifications and a malfunction or setting error.

[0247] Furthermore, as support processing, support information corresponding to the estimated cause can be generated and output. For example, in the case of electronic keyboard instrument specifications, a request message such as "It meets the specifications, what should I do?" can be generated and output.

[0248] In the event of a failure, for example, as support processing, it is possible to perform prompts for the repair center, generate and output link information to the repair center (such as a telephone number, link information to a web page), prompts for estimated repair costs, temporary response instructions (such as instructions to temporarily not use the musical instrument, etc.), etc.

[0249] In the event of a setting error, for example, a link to a webpage with detailed information on electronic keyboard instrument parameter settings can be generated and output as support information. Furthermore, a process image for correcting the setting can be generated and output. For example, a virtual image can be generated as the process image and superimposed on the captured image.

[0250] (Application Example 6: Inquiries regarding the movements of playing a musical instrument)

[0251] Content of inquiry: The keyboard of an electronic keyboard instrument is experiencing abnormal operation (poor rebound).

[0252] Valid images include: Images of a performance (including the sound of the performance), Images of the movement of an instrument (including the sound of the performance), and Images of the movement of a keyboard (including the sound of the performance).

[0253] The current information generated by the current information generating unit 15 includes: information on the structure of the electronic keyboard instrument, such as the type, model, and product number of the electronic keyboard instrument; and information on the sound played by the electronic keyboard instrument.

[0254] The support processing performed by the support processing unit 16 can determine whether the electronic keyboard instrument is malfunctioning. If a malfunction occurs, the cause of the malfunction can be estimated. For example, the cause can be determined to be mechanical damage due to aging, or foreign matter such as a pin that has fallen into the gaps between the keyboard keys.

[0255] Furthermore, the presence information generating unit 15 and the support processing unit 16 can extract and separate the button operation sound as a sound estimated to be effective (important) in solving the consultation matter. By using this button operation sound, more accurate support can be achieved.

[0256] In addition, as support processing, it is possible to perform prompts for repair entrusting places, generate and output link information to repair entrusting places (such as telephone numbers, link information to web pages), prompts for estimated repair costs, temporary response instructions (such as instructions to temporarily not use musical instruments, etc.), etc.

[0257] (Application example 7: Inquiries regarding maintenance of musical instruments)

[0258] Content of inquiry: I would like to know how to maintain my musical instrument.

[0259] Effective captured images: images of musical instruments.

[0260] The current status information generated by the current status information generating unit 15 includes: information related to the structure of the musical instrument, such as the type, model, and product number of the musical instrument; and information related to the status of the musical instrument.

[0261] The support processing performed by the support processing unit 16 can search for FAQ information related to musical instrument maintenance and generate and output as support information links to web pages that provide detailed information on musical instrument maintenance and web pages for purchasing musical instrument maintenance products.

[0262] Furthermore, a process image showing the maintenance process can be generated and output as support information. For example, a process image showing the maintenance process can be superimposed on the musical instrument reflected in the image captured by the performer 3.

[0263] (Application Example 8: Inquiries regarding musical instrument malfunctions)

[0264] Content of inquiry: I would like to dispose of a faulty musical instrument.

[0265] Valid images include: images of the instrument itself, including images of its surrounding equipment, and images of the instrument in motion if it is a movable instrument.

[0266] Input information input by the player 3: detailed information on the disposal method (do you want to discard? Do you want to sell? etc.).

[0267] The status information generated by the status information generating unit 15 includes: information related to the structure of the instrument, such as the type, model, product number, size, and weight of the instrument; information related to the instrument's condition, such as the severity of the malfunction; and information related to the instrument's location, such as where the instrument is installed. Furthermore, status information can also include information on whether the instrument is eligible for a return service or whether it is a model subject to a recall.

[0268] The support processing performed by the support processing unit 16 can generate and output, as support information, links to the pick-up service (e.g., a phone number or a link to a website) for musical instruments that offer a pick-up service. Furthermore, it can also generate and output links to scrap recycling companies, etc. (e.g., a phone number or a link to a website). Furthermore, in the case of instruments subject to recall, it can also generate and output links to designated consultation windows (e.g., a phone number or a link to a website).

[0269] We also support inquiries regarding the disposal of musical instruments, regardless of whether the instrument is malfunctioning or not.

[0270] (Application Example 9: Inquiries regarding the mastery of musical instruments)

[0271] Content of inquiry: I would like to join a music class.

[0272] Effective captured images: images of musical instruments.

[0273] The input information input by the performer 3: the personal information of the consultant. Information related to the course type, such as what kind of course he / she wants to receive.

[0274] The current situation information generated by the current situation information generating unit 15 includes: information related to the structure of the musical instrument, such as the type, model, and product number of the musical instrument; and the age group and residential area of ​​the consultant.

[0275] The support processing performed by the support processing unit 16 allows for searching for music studios that match the inquiring user's desired instrument lessons, age group, and location. Furthermore, it can generate and output links to music studios (e.g., phone numbers or webpages) as support information. Furthermore, it can introduce online courses and educational applications, and generate and output links to webpages for online course attendance.

[0276] [Generation of progress information]

[0277] The support processing unit 16 can also generate progress information related to the progress of support for resolving the consultation. For example, in the communication between the performer 3 and the support staff 4 using the shared screen window, it can be determined whether the consultation is progressing smoothly towards resolution, whether the status is stagnant, or whether the status is deteriorating, and the result can be generated as progress information.

[0278] Furthermore, the work leading to a solution can be divided into stages (steps), and information on which step the current situation is in can be generated as progress information. For example, the stages can be divided into cause identification step, first solution step, second solution step, and solution completion step.

[0279] Furthermore, progress information can be generated, such as the time required to identify the cause (the time required for the cause identification step) and the time required for each step. This allows analysis of steps that are easy for the consulting user to complete and steps that are difficult for the consulting user, enabling efficient support in the future.

[0280] Furthermore, support users can analyze which steps are easy to support and which are difficult to support, allowing them to flexibly utilize this information for efficient future support. Furthermore, support users can analyze which steps they are good at supporting and which steps they are not good at supporting, enabling them to efficiently change the configuration of support staff.

[0281] The generation of progress information may also be automatically performed by the support processing unit 16 of the server device 7 using machine learning or the like. This is not limited to this, and progress information may also be generated based on progress confirmation input from a support-side user such as a support staff member. In the present musical instrument-related support system 1, for example, efficient support using a shared screen window is possible, making it easy to confirm the progress of a consultation until it is resolved, and thus generating highly accurate progress information.

[0282] In this embodiment, the musical instrument association support system corresponds to one embodiment of the information processing system according to the present invention. The server device 7 corresponds to one embodiment of the information processing device according to the present invention.

[0283] As described above, in the musical instrument association support system 1 involved in the present embodiment, a photographic image of the musical instrument 2 and information on consultation matters related to the musical instrument 2 are obtained. In addition, based on the photographic image of the musical instrument, current status information related to the current status of the musical instrument is generated. Based on the information on the consultation matters and the current status information, support information for solving the consultation matters is generated and output. Thus, the resolution of consultation matters related to the musical instrument can be efficiently supported. In particular, in the above-mentioned embodiment, information on consultation matters related to the function of the musical instrument 2 is obtained, and through this musical instrument association support system 1, consultation matters related to the function of the musical instrument 2 can be efficiently resolved.

[0284] Through this musical instrument association support system 1, a system can be constructed to efficiently and appropriately obtain the information required by support side users such as support staff 4 from consulting side users such as performers 3 in the support of musical instruments 2 such as electronic drums. This system can reduce the burden on both consulting side users and support side users, for example, improve the satisfaction of performance activities of performers 3 and others.

[0285] By implementing this technology, in support of faults occurring during the performance of the instrument 2 by the performer 3, the product number / setting status / sound information of the instrument can be automatically obtained from the image of the actual instrument, and the information required for support can be conveyed to the user on the supporting side with high accuracy and efficiency.

[0286] In addition, by implementing this technology, the screen displays instructions from the supporting user, etc., and sends images of the solution operations performed by the consulting user to the supporting user, thereby fully preventing process errors and allowing smooth resumption of performance activities.

[0287] Furthermore, by implementing this technology, when consulting users such as the performer 3 receive support, "unnecessary additional knowledge" and "stress caused by contact" can be reduced, and value such as more focused performance of the instrument can be provided.

[0288] For example, when you feel lost or encounter difficulties playing an instrument, you often can't solve the problem on your own. When you consult support staff, they'll ask you in detail about your current situation, but most exchanges occur over the phone or via text. Therefore, understanding your current situation takes time and effort. Furthermore, implementing the measures instructed by support staff and returning to a good state requires even more time and effort.

[0289] There may be players 3 who, disdaining the hassle of such a burden, continue to use their instrument without contacting support, or even stay away from playing. These players 3 often believe that they should simply focus on playing and that memorizing troubleshooting methods is unnecessary work.

[0290] On the other hand, the places and purposes for which musical instruments 2 are used vary, and the possibility of malfunction cannot be completely eliminated. Therefore, it is difficult to provide a universal musical instrument that is free of malfunctions, etc., or for the musical instrument itself to be 100% resistant to malfunctions. In other words, consultations with consulting users such as players 3 regarding support are still necessary.

[0291] By constructing the present musical instrument-related support system 1 , the burden of receiving support and the burden of resolving consultation matters on the consulting-side user, who is a customer, can be sufficiently reduced, thereby achieving highly efficient support.

[0292] Furthermore, the present musical instrument support system 1 can be constructed with support users as intermediary customers. This system 1 can provide support users who perform support services with information required in the necessary form, thereby reducing their workload and enabling smoother resolution of inquiries.

[0293] For example, when assisting a client who is struggling with playing a musical instrument, the situations on site vary widely. Therefore, it is crucial to investigate the client's current situation at the outset of support. However, the client may not always be able to provide all the information needed for support.

[0294] For example, if you're asked about your electronic drum machine's model name, kit name, firmware, and how long you've been playing it, or the OS version of your smartphone, you might not be able to answer. Similarly, many people wouldn't know the model year of any instrument if asked, for example.

[0295] For example, it's difficult to smoothly obtain this information simply through phone calls or emails. On the other hand, if support users can easily obtain this information during the initial contact phase, they can combine it with technical expertise (such as FAQ information) such as "common problems for each model" to provide more efficient support.

[0296] Furthermore, repeated questions during information delivery are expected to create stress for both support and consulting users. On the other hand, smooth delivery can create a sense of satisfaction, such as "As expected of a manufacturer, they're knowledgeable" or "I'm relieved that they didn't ask unnecessary questions," creating high added value.

[0297] By constructing this musical instrument association support system 1, for example, it is possible to efficiently provide the supporting user with information necessary for support that the consulting user cannot immediately answer or that would be stressful and time-consuming for the consulting user to investigate. Furthermore, it is also possible to efficiently convey information such as instructions from the supporting user to the consulting user. As a result, it is possible to efficiently support the resolution of consulting matters.

[0298] <Second embodiment>

[0299] A second embodiment of the present invention will be described with respect to a musical instrument association support system. In the following description, descriptions of parts having the same configuration and function as those of the musical instrument association support system 1 described in the above embodiment will be omitted or simplified.

[0300] Figure 10 It is a schematic diagram showing a configuration example of a musical instrument association support system according to the second embodiment. Figure 10 The musical instrument association support system 36 shown uses a field support side terminal 37 instead of Figure 1 The consulting terminal 5 is shown.

[0301] The on-site support terminal 37 is a terminal that is used by the on-site Figure 1 The terminal is used by a supporting user (hereinafter referred to as an on-site supporting user) 40 who is face-to-face with a consulting user 39 such as the performer 3 and responds to inquiries from the consulting user 39 .

[0302] exist Figure 10 In the example shown, a smartphone is used as the field support terminal 37, but the present invention is not limited to this and any computer with a camera function can be used. In addition, the field support terminal 37 and the support terminal 6 each have a call function, enabling a field support user 40 to communicate with another support user, the support staff member 4.

[0303] Examples of the on-site support user 40 include on-site staff of a musical instrument manufacturer, music teachers at schools, teachers at music studios, and clerks at musical instrument stores.

[0304] For example, a terminal having the same functions as the consultation terminal 5 described in the first embodiment can be used as the on-site support terminal 37. It is assumed that the on-site support user 40 has accumulated sufficient training in the use of the on-site support terminal 37. In this case, compared to the case where the consultation user 39 uses the consultation terminal 5, operations such as capturing the necessary images of the musical instrument 2, inputting necessary information, and using the shared screen window can be performed more efficiently. As a result, it is possible to efficiently support the resolution of consultation matters.

[0305] Of course, the functions of the field support terminal 37 may be configured differently from those of the consulting terminal 5, such that the terminal is dedicated to the field support user 40. For example, compared to the consulting terminal 5, the terminal may output operating instructions based on more specialized knowledge or provide support using specialized terminology.

[0306] For example, assume that a highly knowledgeable and experienced support staff member handles inquiries on-site. In this scenario, the problem is typically divided, necessary countermeasures are taken, the current situation is understood, and the cause is determined. For example, after the problem is divided, it is assumed that communication with the support staff member 4 via telephone or other means, and web searches via their own terminals, can effectively address the situation on-site. Of course, further utilizing this instrument-related support system 1 can also improve support efficiency.

[0307] There are also cases where the on-site support user 40 is someone with limited knowledge and experience. In this case, it may be difficult to classify the problem and understand the current situation, and information exchange with the support staff 4 may not be smooth. This may lead to frustration for the consulting user 39 on site.

[0308] In the present musical instrument-related support system 36 , the on-site support user 40 using the on-site support terminal 37 can efficiently support solutions to various consultation matters.

[0309] For example, the field support terminal 37 can output current status information and support information, such as the "instrument type," "manufacturer, model," "installation location," and "extent of damage apparent from appearance," which can be determined from the product's appearance. Furthermore, the field support terminal 37 can output search results for support history information and FAQ information.

[0310] Furthermore, for example, when instructions from highly specialized staff are required, a shared screen window can be established with the remote support terminal 6 to display the necessary procedures and operational guidance. This shared screen window allows for visual and intuitive communication of questions and instructions, eliminating the need for extensive use of ambiguous pronouns such as "that" and "this." Furthermore, since the support staff 4 can also see the measures being implemented by the on-site support user 40, they can verify that the instructions were accurately conveyed.

[0311] There may be situations where it is necessary to dispatch an expert to the site. In this case, the on-site support terminal 37 can be used to efficiently transmit initial information to the expert, thereby preventing the need for repeated inquiries. Of course, if the expert uses the support terminal, initial information can be obtained before arriving at the site. These effects also apply to the first embodiment.

[0312] As described above, the musical instrument support system 36 according to this embodiment can also efficiently support the resolution of consultation matters. The on-site support terminal 37 can also be referred to as a "musical instrument identification / guidance terminal" or a "instruction terminal."

[0313] <Third embodiment>

[0314] Figure 11 It is a schematic diagram showing a configuration example of a musical instrument association support system according to a third embodiment. Figure 11 The illustrated instrument association support system 42 is not prepared Figure 1 The supporting terminal 6 is shown. That is, Figure 11 The illustrated musical instrument-related support system 42 is a system in which support services that can be provided by the support staff 4 that can respond remotely are not provided.

[0315] exist Figure 11 In the illustrated musical instrument association support system 42, automatic support is performed by the server device 7. For example, various service processes utilizing machine learning and the output of various service information are automatically executed. Of course, the system can also output search results for support history information and FAQ information.

[0316] The consulting user 39 can input consulting matters at any time 24 hours a day without worrying about time, and can also use the musical instrument association support system 42 without worry even if the user is not good at talking to other people.

[0317] in addition, Figure 11 The illustrated instrument association support system 42 can also be viewed as Figure 1 The musical instrument support system 1 shown is configured to accept customer service of remote support performed by a support staff 4 outside of business hours.

[0318] From that point of view, it can be said that Figure 1 The musical instrument-related support system 1 shown is a useful system that enables efficient support even outside the business hours of customer service.

[0319] The consulting user 39 can also switch the communication mode with the support staff 4 on / off. For example, the communication mode with the support staff 4 is initially set to off, and the communication mode is switched on / off based on the consultation. Figure 11 Then, when the consultation matter seems to be unsolvable or when more knowledge is needed, the communication mode with the support staff 4 is turned on and the support staff 4 is accepted. Figure 1 The illustrated musical instrument association support system 1 supports the following operations.

[0320] exist Figure 10 The illustrated musical instrument support system 36 may be configured without a support terminal 6. In other words, it may be configured without providing support services by remote support staff 4. Of course, it is also possible to switch between customer service during business hours and after hours.

[0321] <Other Implementation Methods>

[0322] The present invention is not limited to the above-described embodiment, and various other embodiments are possible.

[0323] As the support processing by the support processing unit 16 , the following procedures of steps 1 to 3 may be executed.

[0324] Step 1 The consulting user takes a picture of the entire musical instrument 2 that is the subject of the consultation → shares it (for example, displays it in a shared screen window)

[0325] Step 2 The consulting user photographs a portion of the entire (plural) musical instruments 2 → shares (e.g., displays in a shared screen window)

[0326] Step 3 The consulting user operates or plays the musical instrument 2 to produce sound, and the response of the musical instrument 2 is recorded or shared (it is also possible to record the response when it is different from the consulting user's expectation).

[0327] Automatic response to the share in step 1

[0328] Based on the product identification result obtained from the entire captured image of the musical instrument 2, the consulting user is instructed in step 2 to capture (share) a "part" (e.g., the nameplate, the area around the product number, or, in the case of an electronic drum, the area around the sound source module). Product identification can be performed, for example, based on a product photo database containing product photos.

[0329] In addition, in the case of an "assembled object" such as a drum set, there may be cases where it is not assembled as shown in the product photo DB. Therefore, when a drum is included in the photo, a drum identification algorithm can be prepared to perform identification.

[0330] Furthermore, a "marker" or the like can be superimposed on the entire photograph of the musical instrument 2 to indicate the specific location of the "part" to the consulting user (the shooting direction can also be displayed). Furthermore, in digital pianos and other devices, there may be other objects placed on top of the product, such as sheet music or a tablet PC on the music stand. The removal of these other objects can also be incorporated into the judgment algorithm.

[0331] Automatic response to the share in step 2

[0332] If the necessary "partial" captured images are not available, the user is notified of the capture / submission of the captured images. The consulting user is notified of specific instructions for "producing (instrumental instrument) sound" in the next step 3.

[0333] Automatic response to the share in step 3

[0334] As long as the information obtained in steps 1-3 can be used to send the corresponding support history information 20 and FAQ information 23 to the consulting user, this is done. Alternatively, (if it is during customer service hours) a connection is made to a support staff member. If it is outside customer service hours, the information from steps 1-3 is saved so that it can be used by the support staff member for the next support call. The storage destination is not limited.

[0335] This generates supporting progress information, such as whether the problem is progressing smoothly towards resolution. This information is obtained for each of the first through third steps mentioned above. This information can also be flexibly utilized through AI learning. For example, it can be used to determine how long the problem has persisted and how long each step will take to progress.

[0336] As an application, the support method can also be changed. For example, if the user is not used to consulting, the support staff can start to respond from the stage 1 to 3 above instead of automatically responding. This response is also possible.

[0337] This technology can also be used to build an automatic diagnosis system. For example, the response of the instrument 2 can be recorded and analyzed based on the feedback. This can achieve automatic diagnosis such as "Please play this" → "This is not right."

[0338] If the instrument 2 being consulted is an electronic instrument, parameter access (output) and input (input) can also be integrated. Furthermore, if a smart device can be connected to the instrument 2 via wired or wireless communication, the model name and current parameter settings of the instrument 2 can be remotely controlled. Of course, this can also require the consent of the user providing the consultation.

[0339] Various information, including support information exchanged from the acquisition of captured images and information on consultation matters to the resolution of consultation matters, can be accumulated / analyzed and databased. For example, the support history information DB 19 and FAQ information DB 22 can be updated and constructed. In addition, a new fault DB can also be constructed.

[0340] Furthermore, information such as the usage status of consulting users (installation location, performance venue, performance style, and what areas are causing problems) can be accumulated and analyzed as a "usage status database" and flexibly utilized for new product development.

[0341] Can also be based on Figure 5 The user ID as exemplified in can realize the cooperation of purchase history etc. In addition, it is also possible to set that the application using this musical instrument association support system is given priority to receive support.

[0342] In the above embodiment, the method of downloading an application (app) for utilizing the musical instrument association support system 1 is exemplified. Specifically, the method of utilizing the musical instrument association support system 1 using a native application is exemplified. However, this is not a limitation; the musical instrument association support system 1 can also be utilized via a web application, using a web page displayed by a web browser. In this case, the trouble of installing an application is eliminated, and issues such as insufficient terminal storage are avoided.

[0343] In each of the above-mentioned embodiments, the server device 7 functions as one embodiment of the information processing device involved in the present invention. Figure 1 The consulting terminal 5 such as the smartphone shown may also have the function of the server device 7. In this case, the consulting terminal 5 functions as an embodiment of the information processing device involved in the present invention. Figure 1 The support terminal 6, such as a laptop PC, may also have the functions of the server device 7. In this case, the support terminal 6 functions as one embodiment of the information processing device of the present invention. Alternatively, some of the functions of the server device 7 may be installed in the consulting terminal 5 or the support terminal 6.

[0344] Figure 121 is a block diagram showing a hardware configuration example of a computer 60 that can be used as the consultation terminal 5, the support terminal 6, and the server device 7. The consultation terminal 5 also has an imaging function such as a camera.

[0345] The computer 60 includes a CPU 61, a ROM 62, a RAM 63, an input / output interface 65, and a bus 64 interconnecting these. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and the like.

[0346] The display unit 66 is a display device using, for example, liquid crystal, EL, etc. The input unit 67 is, for example, a keyboard, a pointing device, a touch panel, or other operating device. When the input unit 67 includes a touch panel, the touch panel may be integrated with the display unit 66.

[0347] The storage unit 68 is a nonvolatile storage device, such as an HDD, a flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable recording medium 71 such as an optical recording medium or a magnetic recording tape.

[0348] The communication unit 69 is a modem, router, or other communication device that can be connected to a LAN, WAN, or the like and is used to communicate with other devices. The communication unit 69 can communicate using either wired or wireless methods. In many cases, the communication unit 69 is used separately from the computer 60.

[0349] The information processing performed by the computer 60 having the hardware configuration described above is achieved through the cooperation of software stored in the storage unit 68 or ROM 62, etc., and the hardware resources of the computer 60. Specifically, the information processing method of the present invention is achieved by loading the program constituting the software stored in ROM 62, etc. into RAM 63 and executing it.

[0350] The program is installed in the computer 60 via, for example, the recording medium 71. Alternatively, the program may be installed in the computer 60 via a global network or the like. Alternatively, any computer-readable non-transitory storage medium may be used.

[0351] The information processing system and information processing apparatus according to the present invention may be constructed by executing the information processing method (inquiry resolution support method) and program according to the present invention in cooperation with a plurality of computers communicably connected via a network or the like.

[0352] That is, the information processing method and program according to the present invention can be executed not only in a computer system composed of a single computer but also in a computer system in which a plurality of computers operate in conjunction with each other.

[0353] In this disclosure, the term "system" refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all components are housed in the same housing. Thus, multiple devices housed in separate housings and connected via a network, as well as a single device housing multiple modules, are both considered systems.

[0354] Execution of the information processing methods and programs of the present invention by a computer system includes, for example, both cases where a single computer performs the acquisition of photographed images of musical instruments, the acquisition of consultation information, the generation of current status information, the generation and output of support information, the generation and output of questionnaires, and the acquisition of response information, as well as cases where each process is performed by a separate computer. Furthermore, execution of each process by a specified computer includes causing another computer to execute part or all of the process and obtain the results.

[0355] That is, the information processing method and program according to the present invention can also be applied to a cloud computing configuration in which a single function is shared and jointly processed by a plurality of devices via a network.

[0356] The musical instrument association support system, consulting terminal, supporting terminal, server device, various databases, various structures of the GUI for outputting support information, and various processing flows such as obtaining consultation information, generating and outputting support information, etc., described with reference to the accompanying drawings, are merely one embodiment and can be arbitrarily modified without departing from the scope of the present invention. In other words, any other arbitrary structures, algorithms, etc., may be employed to implement the present invention.

[0357] In this disclosure, terms such as "approximately," "substantially," and "approximately" are sometimes used as appropriate to facilitate understanding. However, there is no clear distinction between the use and non-use of terms such as "approximately," "substantially," and "approximately."

[0358] That is, in the present disclosure, concepts such as "center", "central", "uniform", "equal", "same", "orthogonal", "parallel", "symmetrical", "extended", "axial direction", "cylindrical shape", "cylindrical shape", "annular shape", "annular shape" and other specified shapes, sizes, positional relationships, states, etc. are set to include concepts such as "substantially center", "substantially central", "substantially uniform", "substantially equal", "substantially the same", "substantially orthogonal", "substantially parallel", "substantially symmetrical", "substantially extended", "substantially axial direction", "substantially cylindrical shape", "substantially cylindrical shape", "substantially annular shape", "substantially annular shape" and the like.

[0359] For example, it also includes states within a specified range (for example, a range of ±10%) based on "completely centered", "completely central", "completely uniform", "completely equal", "completely identical", "completely orthogonal", "completely parallel", "completely symmetrical", "completely extended", "completely axial direction", "completely cylindrical", "completely cylindrical", "completely annular", "completely circular", etc.

[0360] Therefore, even when the terms "approximately," "roughly," or "about" are not added, concepts expressed with the terms "approximately," "roughly," or "approximately" can be included. Conversely, states expressed with the terms "approximately," "roughly," or "about" do not necessarily exclude complete states.

[0361] In this disclosure, expressions using the word "relative to" such as "larger than A" and "smaller than A" are intended to encompass both concepts that include the equivalent to A and concepts that exclude the equivalent to A. For example, "larger than A" is not limited to excluding the equivalent to A and also includes "greater than A." Furthermore, "smaller than A" is not limited to "less than A" and also includes "less than A."

[0362] When implementing the present technology, specific settings and the like may be appropriately adopted according to the concepts encompassed by “larger than A” and “smaller than A” so as to exhibit the effects described above.

[0363] It is also possible to combine at least two of the characteristic features of the present invention described above. That is, the various characteristic features described in each embodiment can be arbitrarily combined without distinguishing between the respective embodiments. In addition, the various effects described above are merely illustrative and not limiting, and other effects may also be exerted.

[0364] For example, the following configuration examples can be cited as the information processing system, information processing method, program, and information processing device according to the present invention.

[0365] (1) An information processing system comprising:

[0366] a first acquiring unit, acquiring an image of the musical instrument;

[0367] a second acquiring unit, acquiring information on consultation matters related to the photographed musical instrument;

[0368] a current information generating unit that generates current information related to the current status of the musical instrument based on the captured image, the current information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and

[0369] The support processing unit generates and outputs support information for solving the consultation matter based on the information on the consultation matter and the current situation information.

[0370] (2) The information processing system according to (1), wherein:

[0371] The second acquisition unit acquires information on consultation matters related to the function of the musical instrument.

[0372] (3) The information processing system according to (2), wherein:

[0373] The support processing unit generates and outputs a support image for solving the consultation matter.

[0374] (4) The information processing system according to (3), wherein:

[0375] The support processing unit superimposes the support image on the captured image or the model image of the musical instrument and outputs the superimposed image.

[0376] (5) The information processing system according to (2), wherein:

[0377] The support processing unit generates and outputs photography guide information for guiding photography of the musical instrument.

[0378] (6) The information processing system according to (2), further comprising:

[0379] The storage unit stores information including the captured image, the information on the consultation matters, the current situation information, and the support information as support history information.

[0380] (7) The information processing system according to (2), wherein:

[0381] The first acquisition unit acquires, as the captured image, at least one of a performance image of a performer playing the musical instrument and an image including peripheral devices used when playing the musical instrument.

[0382] (8) The information processing system according to (2), wherein:

[0383] The information related to the structure of the musical instrument includes at least one of a type, a model, a product number, and a year.

[0384] (9) The information processing system according to (2), wherein:

[0385] The information related to the status of the musical instrument includes at least one of a setting status and a parameter setting status.

[0386] (10) The information processing system according to (2), wherein:

[0387] The information on the installation location of the musical instrument includes information on whether the musical instrument is installed indoors or outdoors.

[0388] (11) An information processing method executed by a computer system, the computer system performing the following operations:

[0389] Obtaining images of musical instruments;

[0390] Obtain information on consultation matters related to the photographed musical instrument;

[0391] generating, based on the captured image, current status information related to the current status of the musical instrument, the current status information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and

[0392] Based on the information about the consultation matter and the current situation information, support information for solving the consultation matter is generated and output.

[0393] (12) A program that causes a computer system to:

[0394] Obtaining images of musical instruments;

[0395] Obtain information on consultation matters related to the photographed musical instrument;

[0396] generating, based on the captured image, current status information related to the current status of the musical instrument, the current status information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and

[0397] Based on the information about the consultation matter and the current situation information, support information for solving the consultation matter is generated and output.

[0398] (13) An information processing device comprising:

[0399] a first acquiring unit, acquiring an image of the musical instrument;

[0400] a second acquiring unit, acquiring information on consultation matters related to the photographed musical instrument;

[0401] a current information generating unit that generates current information related to the current status of the musical instrument based on the captured image, the current information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and

[0402] The support processing unit generates and outputs support information for solving the consultation matter based on the information on the consultation matter and the current situation information.

[0403] Description of Reference Numerals

[0404] 1, 36, 42: Musical instrument association support system, 2: Musical instrument, 3: Performer, 4: Support staff, 5: Consultation side terminal, 6: Support side terminal, 7: Server device, 11: DB, 19: Support history information DB, 20: Support history information, 22: FAQ information DB, 23: FAQ information, 26, 29: Virtual image, 37: On-site support side terminal, 39: Consultation side user, 40: On-site support side user.

Claims

1. An information processing system comprising: a first acquiring unit, acquiring an image of the musical instrument; a second acquiring unit, acquiring information on consultation matters related to the function of the photographed musical instrument; a current information generating unit that generates current information related to the current status of the musical instrument based on the captured image, the current information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and The support processing unit generates and outputs support information for solving the consultation matter based on the information on the consultation matter and the current situation information.

2. The information processing system according to claim 1, wherein: The information processing system further includes: a storage unit for storing at least one of support history information and FAQ information, wherein the support history information includes information on consultation matters obtained in the past and support information generated and output in the past to solve the consultation matters, and the FAQ information includes information on consultation matters and support information created in advance, The support processing unit searches for at least one of the stored support history information and the FAQ information, and outputs a search result.

3. The information processing system according to claim 1, wherein: The information on the consultation matters includes at least one of consultation matters related to the pronunciation of the musical instrument, consultation matters related to the movements of the musical instrument during performance, consultation matters related to the settings of the musical instrument, consultation matters related to the malfunction of the musical instrument, consultation matters related to the maintenance of the musical instrument, and consultation matters related to the extended functions of the musical instrument.

4. The information processing system according to claim 1, wherein: The captured image contains sound information, The first acquisition unit acquires information on the musical instrument's performance sound. At least one of the current situation information generation unit generates the current situation information based on the acquired information about the musical performance, and the support processing unit generates the support information based on the acquired information about the musical performance.

5. The information processing system according to claim 1, wherein: The support processing section generates and outputs a process image for indicating a process to be performed on the musical instrument.

6. The information processing system according to claim 1, wherein: The information processing system further comprises: The questionnaire unit outputs a questionnaire related to the consultation matter and obtains answer information to the questionnaire. At least one of generation of the presence information by the presence information generating unit based on the acquired answer information and generation of the support information by the support processing unit based on the acquired answer information is executed.

7. The information processing system according to claim 1, wherein: The current situation information generating unit generates the current situation information based on the information on the consultation matter.

8. The information processing system according to claim 1, wherein: The support processing unit generates and outputs information indicating at least one of a photographing location, a number of photographs, and a photographing direction of the musical instrument as photographing guide information for guiding photographing of the musical instrument.

9. The information processing system according to claim 1, wherein: The support processing unit generates progress information related to the progress of support for resolving the consultation matter.

10. The information processing system according to claim 1, wherein: The support processing unit outputs link information to a valid web page in order to resolve the consultation matter.

11. The information processing system according to claim 1, wherein: The support processing unit outputs instruction information for instructing performance of the musical instrument.

12. The information processing system according to claim 1, wherein: The second acquisition unit acquires the information on the consultation matter based on at least one of input information input by a user and the captured image.

13. An information processing method executed by a computer system, the computer system performing the following operations: Obtaining images of musical instruments; Obtaining information on consultation matters related to the function of the photographed musical instrument; generating, based on the captured image, current status information related to the current status of the musical instrument, the current status information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and Based on the information about the consultation matter and the current situation information, support information for solving the consultation matter is generated and output.

14. A program that causes a computer system to perform the following operations: Obtaining images of musical instruments; Obtaining information on consultation matters related to the function of the photographed musical instrument; generating, based on the captured image, current status information related to the current status of the musical instrument, the current status information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and Based on the information about the consultation matter and the current situation information, support information for solving the consultation matter is generated and output.

15. An information processing device comprising: a first acquiring unit, acquiring an image of the musical instrument; a second acquiring unit, acquiring information on consultation matters related to the function of the photographed musical instrument; a current information generating unit that generates current information related to the current status of the musical instrument based on the captured image, the current information including information related to at least one of the structure of the musical instrument, the state of the musical instrument, and the installation position of the musical instrument; and The support processing unit generates and outputs support information for solving the consultation matter based on the information on the consultation matter and the current situation information.

Images