Data output device and data output method

Abstract

To provide a data output device capable of generating nonverbal information data desired by a user on the basis of language information.SOLUTION: A data output device 100 comprises: an input unit (input device 191) which acquires input information entered by a user as linguistically expressed information; a generation unit 113 which generates data corresponding to the linguistically expressed information acquired by the input unit as nonverbal information on the basis of user attributes; and an output unit (output device 192) which outputs the generated data in a form perceptible through at least one of five human senses. Furthermore, the data output device 100 may also comprise an evaluation result acquisition unit 114 which acquires user's evaluation results for the output results, and an update unit 115 which repeatedly instructs the generation unit 113 to generate data until the evaluation results of the output results of data corresponding to the input information where the evaluation results are reflected become the evaluation results matching the user's desired output results.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a data output device and a data output method that generate and output data corresponding to input information indicated by a language.

Background Art

[0002] With the declining birthrate and aging population, there is a shortage of labor, and the transfer of skills from skilled workers to unskilled workers has become an issue. For example, in visual inspections and sound inspections in equipment or facility inspections, abnormal conditions or signs indicating abnormalities may be indicated by a language (words, language information, language expressions) such as characters or speech. However, it is difficult for unskilled workers with little experience in dealing with equipment / facility abnormalities to actually determine the presence or absence of abnormalities based on language information.

[0003] As a technology related to inspection work using sound, there is an elevator inspection system described in Patent Document 1. This elevator inspection system is an elevator inspection system for inspecting an elevator, and includes a data processing unit that classifies the type of sound generated in the elevator while the inspection is being performed, an input unit that receives input of information regarding the elevator, and onomatopoeia representing the sound during the period when the sound is generated, and a search unit that searches a search database for maintenance information indicating the necessity of maintenance for the elevator based on the information regarding the elevator, the type of the sound, and the onomatopoeia during the period when the sound is generated, and a display unit that displays the maintenance information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The elevator inspection system searches for and displays maintenance information based on onomatopoeia indicating abnormal sounds. However, even for the same abnormal sound, the onomatopoeia indicating it varies among the inspection workers, and appropriate maintenance information is not always displayed. Image data and sound data that serve as samples of abnormal states and signs indicating abnormalities are helpful, but the accumulation of such data is small and it is also difficult to collect. It is desirable to be able to refer to data of sounds or appearances that indicate abnormalities or are signs of abnormalities by some method. This applies not only to sounds and appearances, but also to smells, tastes, tactile sensations, vibrations, etc. that can be sensed by at least one of the five human senses. The present invention has been made in view of such a background, and an object thereof is to provide a data output device and a data output method that enable generation of data of non-verbal information desired by a user based on language information.

Means for Solving the Problems

[0006] In order to solve the above-described problems, a data output device according to the present invention includes an input unit that acquires input information input as information indicated by a language by a user, and data corresponding to the input information indicated by the language acquired by the input unit is generated as data of non-verbal information based on a user attribute that is an attribute of the user, and an output unit that outputs the generated data so that it can be sensed by at least one of the five human senses.

Effects of the Invention

[0007] According to the present invention, it is possible to provide a data output device and a data output method that enable generation of data of non-verbal information desired by a user based on language information. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Modes for Carrying Out the Invention

[0009] ≪Overview of Data Output Device≫ The data output device in the following embodiment for implementing the present invention will be described. The data output device generates and outputs data corresponding to input information represented in a language including onomatopoeia and mimetic words using a machine learning model. In other words, the data output device acquires language information and outputs it as non-language information based on user attributes. The data output device receives the user's evaluation result regarding the output result of the data output corresponding to the input information, generates data corresponding to the input information reflecting the evaluation result, and outputs it again. The data output device repeats receiving the evaluation result and outputting the data until the data desired by the user is obtained. Next, the data output device adjusts the parameters of the machine learning model so that the data corresponding to the first input information becomes the data desired by the user. The parameters (user attributes) of the adjustment result will be different for each user.

[0010] According to such a data output device, for example, regarding sounds indicating abnormalities or signs of abnormalities, even if the input information and evaluation results are different for each user (inspection worker), it becomes possible to generate data of sounds indicating abnormalities / signs of abnormalities desired by the user. By using a data output device with parameters adjusted for each user, it becomes possible to efficiently generate (reducing the repetition of data output and output evaluation) data of sounds and appearances desired by the user corresponding to the language (input information) indicated by the user. By skilled workers using the data output device, it becomes possible to generate data of sounds indicating the original abnormalities and sounds that are signs of abnormalities, enabling the transfer of skills from skilled workers to unskilled workers.

[0011] Hereinafter, a data output device related to the inspection of facilities and equipment will be described. A user inputs, via a keyboard or the like, as text or as voice toward a microphone, a sound indicating an abnormality or a sound indicating a sign of an abnormality (such as a sound in a knocking inspection or a sound during operation). The data output device generates and outputs sound data corresponding to the input information, which is this language information (text or voice), or image data indicating an appearance (such as discoloration, rust, or deformation). A data output device that generates data in output formats (modalities) such as smell, taste, touch, and vibration, or a data output device that generates data in a plurality of modalities can be implemented in the same manner. Note that text input may sometimes be the conversion of voice input via a microphone into text.

[0012] <<First Embodiment: Usage Procedure of Data Output Device>> Before explaining the configuration of the data output device 100 (see FIG. 2 described later) according to the first embodiment, the usage procedure of the data output device 100 will be explained. FIG. 1 is a flowchart of the usage procedure of the data output device 100 according to the first embodiment.

[0013] In step S11, the user (inspection worker) inputs a language expression of a desired sound to the data output device 100. For example, the user inputs, as an expression, a sound indicating an abnormality of a facility / equipment as "a sound like clank". In response to this input, the data output device 100 outputs a sound corresponding to the input. Here, the input is text information manually input by the user or voice information uttered by the user, which is language information. Here, the output is a sound from a speaker that can be sensed by at least one of the five senses of a human, which is non-language information.

[0014] In step S12, the user evaluates the output sound. For example, if the output sound is low, the user evaluates that the desired sound is a higher sound. If the output sound is sufficiently similar to the desired sound, the user evaluates it as OK. In step S13, if the evaluation result in step S12 is OK (step S13 → YES), the user proceeds to step S14, and otherwise (step S13 → NO), the user proceeds to step S15.

[0015] In step S14, the user inputs "OK" to the data output device 100. In step S15, the user inputs an evaluation result (e.g., "higher sound") to the data output device 100 and returns to step S12. The user will repeat the evaluation of the sound output by the data output device 100 (see step S12) until the desired sound is output.

[0016] ≪First Embodiment: Configuration of Data Output Device≫ Figure 2 is a functional block diagram of the data output device 100 according to the first embodiment. The data output device 100 is a computer and includes a control unit 110, a storage unit 120, and an input / output unit 180. An input device 191 (input unit) such as a keyboard, a mouse, and a microphone and an output device 192 (output unit) such as a display and a speaker are connected to the input / output unit 180. The input / output unit 180 may include a communication device and be capable of data transmission and reception with other input devices and output devices. Note that the data output device 100 may include the input device 191 or the output device 192.

[0017] ≪First Embodiment: Storage Unit≫ The storage unit 120 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 120 stores a data generation model 130, a user attribute database 140, a user information database 150, and a program 128. The program 128 includes a description of data output processing (see Figure 4) described later.

[0018] ≪First Embodiment: Data Generation Model≫ The data generation model 130 is a machine learning model, such as a neural network or a deep learning model. The data generation model 130 uses input information represented by a language (e.g., text) as an explanatory variable and sound data corresponding to the input information as an objective variable. Such a data generation model 130 can be generated, for example, by training (learning) using training data with text as an explanatory variable and sound data corresponding to the text as a correct label. It may also be a machine learning model using a generative AI technique that takes prompt information in the form of text as input and generates sound data or image data corresponding to the prompt information.

[0019] ≪First Embodiment: User Attribute Database · User Information Database≫ In the user attribute database 140, the user (inspection operator) of the data output device 100 and the user attributes of the user are stored in an associated manner. Note that the user attribute is the correspondence between the sound desired by the user and the language expression indicating the sound. The user attribute in the first embodiment is a parameter of the data generation model 130 (see the parameters 621 and 622 in FIG. 3 described later), and is, for example, the weight of a neural network.

[0020] In the user information database 150, the user and the user information of the user are stored in an associated manner. Examples of user information include gender, age, number of years of inspection work experience, equipment / devices with inspection work experience, etc. The user information may include sensory inspection results of the five senses, such as auditory diagnosis inspection results and visual inspection results (e.g., color vision inspection results).

[0021] ≪First Embodiment: Control Unit≫ The control unit 110 is configured to include a CPU (Central Processing Unit), and is provided with an input acquisition unit 111, an attribute acquisition unit 112, a generation unit 113, an evaluation result acquisition unit 114, and an update unit 115. The control unit 110 may be configured to include a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and the like. FIG. 3 is a diagram for explaining the data output process according to the first embodiment. Hereinafter, the processes of each functional unit will be described with reference to FIG. 3.

[0022] <<First Embodiment: Input Acquisition Unit>> The input acquisition unit 111 acquires the language expression (language information) input via the input device 191 as input information 611. For example, the input acquisition unit 111 acquires, as the input information 611, the text input using a keyboard or the sound "a loud noise" (generated by the user) input using a microphone.

[0023] <<First Embodiment: Attribute Acquisition Unit>> The attribute acquisition unit 112 acquires the parameter 621 of the data generation model 130, which is a user attribute related to the user, from the user attribute database 140. In the case of a new user, the specified parameter 621 of the data generation model 130 is acquired as the user attribute.

[0024] <<First Embodiment: Generation Unit>> The generation unit 113 generates sound data 631 (non-language information) corresponding to the input information 611 using the data generation model 130. Although the data generation model 130 itself is data, it may be described as a functional unit in some cases. For example, it may be described that the data generation model 130 outputs the sound data 631 corresponding to the input information 611.

[0025] As described above, the data output device 100 includes an input unit (see the input device 191) that acquires input information input as information indicated by a language by the user. The data output device 100 includes a generation unit 113 that generates data (see the sound data 631) corresponding to the input information 611 indicated by a language, which is acquired by the input unit, as non-verbal information data (see the sound data 631) based on user attributes (see the parameter 621) that are attributes of the user. The data output device 100 includes an output unit (see the output device 192) that outputs the generated data so that it can be sensed by at least one of the five senses of a human. The generation unit 113 uses a machine learning model (see the data generation model 130) that has been learned to output data corresponding to the input information 611, with the input information 611 indicated by a language as an explanatory variable, and generates data corresponding to the input information 611. The user attributes are the parameters 621 and 622 of the machine learning model.

[0026] ≪First Embodiment: Evaluation Result Acquisition Unit≫ The evaluation result acquisition unit 114 acquires the evaluation result 612 of the user with respect to the sound output from the output device 192 for the sound data 631. If the sound desired by the user is a louder "kankan" sound than the output sound, the evaluation result acquisition unit 114 acquires the evaluation result 612 of "a louder sound" via the input device 191.

[0027] ≪First Embodiment: Update Unit≫ The update unit 115 updates the parameter 621, which is a user attribute, as follows and stores it in the user attribute database. First, the update unit 115 generates input information 613, which is language information reflecting the evaluation result 612, for the first input information 611 input from the input device 191. For example, the update unit 115 generates input information 613 of "a louder kankan sound" by reflecting "a louder sound" on "the kankan sound".

[0028] Next, the update unit 115 instructs the generation unit 113 to obtain the sound data 632 corresponding to the input information 613 and outputs it from the output device 192. Subsequently, the update unit 115 instructs the evaluation result acquisition unit 114 to obtain the user's evaluation result 614. The update unit 115 repeats the generation of the input information 613, the acquisition and output of the sound data 632, and the acquisition of the evaluation result 614 until the sound data for the first input information 611 desired by the user is generated (see steps S12, S13, and S15 in FIG. 1).

[0029] The description will continue assuming that the sound data 632 is the sound data for the first input information 611 desired by the user (for example, the sound of a percussion inspection indicating an abnormality). The update unit 115 adjusts the parameter 621 so that the sound data 633 corresponding to the first input information 611 generated by the generation unit 113 becomes the sound data 632, and stores the parameter 622, which is the adjustment result, in the user attribute database 140. Here, the fact that the sound data 633 becomes the sound data 632 means that the difference between the sound data 632 and 633 as sound data is equal to or less than a predetermined value. As an example of a method for adjusting the parameter 622, there is a gradient method for an optimization problem that searches for the minimum value of an objective function that calculates the difference between the sound data 632 and 633 using the parameter 622 as an input parameter and the difference as a function value. The update unit 115 may adjust the parameter 622 using other methods.

[0030] As described above, the data output device 100 includes an evaluation result acquisition unit 114 that acquires the user's evaluation results 612 and 614 for the output results output by the output unit (see the output device 192). The data output device 100 includes an update unit 115 that repeatedly instructs the generation unit 113 to generate data until the evaluation result 614 of the output result of the data corresponding to the input information 613 reflecting the evaluation result 612 becomes the evaluation result that the output result is the desired output result of the user. The update unit 115 updates the user attribute (see the parameter 622) so that the data (see the sound data 633) corresponding to the input information 611 acquired by the input unit (see the input device 191) becomes the data (see the sound data 632) corresponding to the desired output result.

[0031] <<First Embodiment: Data Output Processing>> FIG. 4 is a flowchart of the data output processing according to the first embodiment. With reference to FIG. 4, a process in which the data output device 100 acquires a linguistic expression of a sound desired by the user and outputs corresponding sound data will be described. In step S21, the input acquisition unit 111 acquires a linguistic expression of the user for the output target data (for example, "the sound of kan-kan"). Note that step S21 is a process corresponding to step S11 shown in FIG. 1.

[0032] In step S22, the generation unit 113 generates and outputs sound data corresponding to the linguistic expression using the data generation model 130. If the output sound is not the desired sound, the user inputs to the data output device 100 via the input device 191 what kind of sound the desired sound is (see step S15 shown in FIG. 1). If the output sound is the desired sound, the user inputs "OK" (see step S14). In step S23, the evaluation result acquisition unit 114 acquires the user's evaluation result for the sound data output in step S22. In step S24, if the evaluation result is the desired sound data 632 (step S24 → YES), the update unit 115 advances the process to step S25; otherwise (step S24 → NO), the update unit 115 advances the process to step S26.

[0033] In step S25, the update unit 115 adjusts the parameter 621 so that the sound data 633 corresponding to the linguistic expression acquired in step S21 generated by the generation unit 113 (see the input information 611 shown in FIG. 3) becomes the desired sound data 632. Next, the update unit 115 stores the adjusted parameter 622 as a user attribute corresponding to the user in the user attribute database 140. In step S26, the update unit 115 generates linguistic information reflecting the evaluation result 612 (for example, "a higher-pitched sound of kan-kan") and returns to step S22.

[0034] <<First Embodiment: Features of Data Output Device>> The data output device 100 acquires the first language expression (input information 611) of the sound desired by the user and the evaluation result 612 for the output sound, and acquires the desired sound data 632. Next, the data output device 100 adjusts the parameters 621 of the data generation model 130 so that the desired sound data 632 can be output from the first language expression, and acquires the parameters 622 which are the adjustment result.

[0035] Even for the same sound, the language expression for that sound varies from user to user. By using the data output device 100 in which the parameters 622 (user attributes) are adjusted for each user, it becomes possible to efficiently generate sound data according to the language expressions of individual users (reducing the repetition of steps S12, S13, and S15 shown in FIG. 1).

[0036] <<Second Embodiment>> The data output device 100 of the first embodiment adjusts the parameters of the data generation model 130 for each user as user attributes (see step S25 shown in FIG. 4). In the data generation model 130A in the second embodiment (see FIGS. 5 and 6 described later), latent variables 661 and 662 which become inputs (explanatory variables) instead of the parameters are used as user attributes.

[0037] FIG. 5 is a functional block diagram of the data output device 100A according to the second embodiment. Compared with the data output device 100 according to the first embodiment (see FIG. 2), the update unit 115A and the data generation model 130A are different. The input (explanatory variable) of the data generation model 130A is a latent variable indicating the language expression of the desired sound and the user attribute which is the correspondence between the sound desired by the user and the language expression indicating the sound. The input of the data generation model 130A may be the distribution of the latent variable instead of the latent variable. The update unit 115A updates the latent variable serving as the input instead of the parameters 621 of the data generation model 130 (see FIG. 3).

[0038] FIG. 6 is a diagram for explaining data output processing according to the second embodiment. The latent variable 661, which is an input when the generation unit 113 generates the sound data 671 corresponding to the first input information 651, is a predetermined latent variable. In other words, the same sound data 671 is generated for the same input information 651 regardless of the user. Not limited to the predetermined latent variable, a random latent variable 661 according to a predetermined probability distribution may be used as the initial latent variable.

[0039] Similar to the first embodiment, the update unit 115A repeats the generation of the input information 653, the acquisition and output of the sound data 672, and the acquisition of the evaluation result 654 until the sound data (non-verbal information) corresponding to the first input information 651 (language information) desired by the user is generated. When the sound desired by the user is obtained, the update unit 115A adjusts the latent variable 661 so that the sound data 673 corresponding to the first input information 651 generated by the generation unit 113 becomes the sound data 672, and obtains the latent variable 662, which is the adjustment result.

[0040] As described above, the generation unit 113 uses the learned machine learning model (see the data generation model 130A) with the input information 651 indicated by language and the latent variables 661, 662 indicating user attributes or the distribution of latent variables as explanatory variables, and generates data corresponding to the input information 651.

[0041] ≪Second Embodiment: Features of Data Output Device≫ The data output device 100A may adjust the latent variable 662 that is an input to the data generation model 130A instead of adjusting the parameter 622 of the data generation model 130 for each user. The data output device 100A can efficiently generate sound data according to the language expression of each user.

[0042] ≪Third Embodiment≫ The data output devices 100 and 100A first acquire the linguistic expression of the sound desired by the user (see step S21), and repeat the evaluation of the output sound (see steps S22 to S24 and S26). After the desired sound is obtained (see step S24 → YES), the data output devices 100 and 100A adjust the parameters 621 and latent variables 661 that are user attributes, and acquire the adjusted parameters 622 and latent variables 662 (see step S25). Instead of this, the data output device 100B according to the third embodiment (see FIG. 7 described later) receives the evaluation result for the sample sound, and adjusts the user attributes so that the sample sound is output when the evaluation result is input.

[0043] FIG. 7 is a functional block diagram of the data output device 100B according to the third embodiment. Compared with the data output device 100 according to the first embodiment, the update unit 115B is different, and sample data 121 is added to the storage unit 120. The sample data 121 is sound data 731, 733, 735 (see FIG. 8 described later) used for adjusting user attributes (parameters of the data generation model 130). The user inputs evaluation results 711, 713, 715 that express the output sound of the sound data in language.

[0044] FIG. 8 is a diagram for explaining the user attribute update process according to the third embodiment. The processing content of the update unit 115B will be described with reference to FIG. 8. For the evaluation result 711 of the user for the output sound of the sound data 731, the update unit 115B adjusts the parameters 721 (user attributes) of the data generation model 130 so that the sound data 732 generated by the generation unit 113 becomes the sound data 731 when the evaluation result 711 is used as the input information 712. For example, if the evaluation result 711 of the output sound of the sound data 731 is "KANッ", the parameters 721 are adjusted so that the sound data 732 when "KANッ" is used as the input information 712 becomes the sound data 731 (so that the difference as sound data is equal to or less than a predetermined value).

[0045] Hereinafter, the update unit 115B repeats this process. Regarding the user evaluation result 713 for the output sound of the sound data 733, the update unit 115B adjusts the parameter 722 so that the sound data 734 generated by the generation unit 113 when the evaluation result 713 is used as the input information 714 becomes the sound data 733. Further, regarding the user evaluation result 715 for the output sound of the sound data 735, the update unit 115B adjusts the parameter 723 so that the sound data 736 generated by the generation unit 113 when the evaluation result 715 is used as the input information 716 becomes the sound data 735. In this way, the update unit 115B repeats the adjustment of the parameters 721, 722, and 723, which are user attributes. Note that the update unit 115B adjusts the parameters of the data generation model 130, but may adjust the latent variables that are the input of the data generation model 130A according to the second embodiment.

[0046] FIG. 9 is a flowchart of the user attribute update process according to the third embodiment. In step S31, the update unit 115B starts a process of repeating steps S32 to S34 for each sample data 121. Hereinafter, the target of the repeating process is referred to as the processing target sample data (sound data). In step S32, the update unit 115B outputs the processing target sample data to the output device 192.

[0047] In step S33, the evaluation result acquisition unit 114 acquires the user evaluation result for the output (output sound) in step S32. In step S34, the update unit 115B adjusts the parameters of the data generation model 130 so that the sound data generated by the generation unit 113 using the evaluation result as the input becomes the processing target sample data (so that the difference as sound data is equal to or less than a predetermined value).

[0048] As described above, the evaluation result acquisition unit 114 acquires information (see evaluation results 711, 713, and 715) indicated in the language corresponding to the output result of the sample data (see sound data 731, 733, and 735). The update unit 115B updates the user attributes (parameters 721, 722, 723) so that the data corresponding to the information (see audio data 732, 734, 736) becomes sample data (see audio data 731, 733, 735).

[0049] ≪Third Embodiment: Features of Data Output Device≫ The data output device 100B adjusts the parameters of the data generation model 130, which is a user attribute, without the user repeatedly inputting or evaluating the output sound in the language expression of the desired sound as shown in FIG. 1. By adjusting the user attributes using the sample data 121 before generating the audio data of the desired sound, it can be expected that the audio data of the desired sound will be generated efficiently.

[0050] ≪Fourth Embodiment≫ The data output device 100B according to the third embodiment adjusts the user attributes of a new user using the sample data 121 (see FIG. 7). The user attributes of a new user may be adjusted based on user information. FIG. 10 is a functional block diagram of the data output device 100C according to the fourth embodiment. The update unit 115C is different from the data output device 100 according to the first embodiment.

[0051] Specifically, the update unit 115C acquires a user whose user attributes have been adjusted and the user attributes have been stored in the user attribute database 140. Next, the update unit 115C acquires the user information of the user from the user information database 150. Subsequently, the update unit 115C sets the user attributes of the user corresponding to the user information that is most similar to the user information of the new user as the user attributes of the new user.

[0052] The update unit 115C may set the average value of the user attributes of a predetermined number of users corresponding to user information similar to the user information of the new user as the user attributes of the new user. Further, the update unit 115C may set the average value of the user attributes weighted according to the similarity as the user attributes of the new user.

[0053] Furthermore, the update unit 115C may perform user attribute update processing (see FIG. 9) according to the third embodiment, using the user attributes (parameters of the data generation model 130 and latent variables which are inputs to the data generation model 130A) thus obtained as initial values. Also, instead of the parameter 621 (see FIG. 3) and the latent variable 661 (see FIG. 6), user attributes based on the user attributes of existing users may be used as the initial values of the user attributes in the data output processing (see FIG. 4).

[0054] As described above, the update unit 115C calculates the user attributes of a new user based on the user attributes of a user who has user information similar to the user information of the new user and who has updated (stored in the user attribute database 140) user attributes.

[0055] ≪Fourth Embodiment: Features of Data Output Device≫ By using, as the user attributes of a new user, the user attributes of a user whose gender, age, years of inspection work experience, and sensory inspection results of the five senses are similar, it is expected that desired sound data can be obtained more efficiently (by reducing the number of evaluations of the output sound) than when using default user attributes.

[0056] ≪Fifth Embodiment≫ The data output devices 100, 100A, 100B, and 100C as inspection support devices will be described below. FIG. 11 is a functional block diagram of a data output device 100D according to the fifth embodiment. Compared with the data output device 100 (see FIG. 2) according to the first embodiment, an inspection work support unit 116 is added to the control unit 110, and an inspection work database 160 is added to the storage unit 120.

[0057] Stored in the inspection work database 160 are information related to inspection work for facilities and equipment, such as the names of facilities and equipment and their identification information, the names of parts to be inspected, their locations, inspection contents, presence or absence of abnormalities, details of abnormalities, inspection dates and times, and inspection workers.

[0058] When an abnormality is found during inspection, the inspection work support unit 116 stores data indicating the abnormality in the inspection work database 160. The data indicating an abnormality includes abnormal sounds of facilities and equipment, abnormal sounds in impact sound inspections, and image data obtained by photographing the part where the abnormality was found.

[0059] When no abnormality is found during inspection, the inspection work support unit 116 generates data indicating an abnormality according to the instructions of the inspection worker who is the user and stores the data in the inspection work database 160. For example, the inspection work support unit 116 first accepts "a clanging sound" as the linguistic expression of an abnormal sound in an impact sound inspection, and after repeatedly receiving the user's evaluation, stores the sound data of the desired abnormal sound in the inspection work database 160 as an example of an abnormal sound.

[0060] As described above, the input information indicates an abnormality of an object (for example, a facility, a machine, or a building / construction as described later) or a sign of an abnormality. The data output device 100D includes an inspection work support unit 116 that stores the generated data as abnormal data related to the object.

[0061] FIG. 12 is a flowchart of the inspection work support process according to the fifth embodiment. The inspection work support process is a process that is repeated for each inspection work target such as a facility, a device, and its part. In step S51, the inspection work support unit 116 displays an inspection instruction including the name, location, inspection content, etc. of the inspection work target. The inspection worker who is the user performs the inspection work according to the inspection instruction and inputs the inspection result.

[0062] In step S52, the inspection work support unit 116 receives the inspection result input by the user. In step S53, if the inspection result is abnormal (step S53→YES), the process proceeds to step S54, and if there is no abnormality (step S53→NO), the process proceeds to step S55. In step S54, the inspection work support unit 116 stores data indicating an abnormality and the details of the abnormality in the inspection work database 160, and ends the inspection work support process.

[0063] In step S55, the inspection work support unit 116 stores data indicating normality in the inspection work database 160. For example, the inspection work support unit 116 stores image data obtained by photographing parts of equipment and devices without abnormalities. In step S56, the inspection work support unit 116 asks the user whether to create a sample of the abnormality or not. If the user answers to create it (step S56 → YES), the process proceeds to step S57. If the inspection work support unit 116 receives an answer not to create it (step S56 → NO), it ends the inspection work support process.

[0064] In step S57, the inspection work support unit 116 generates data of a sample indicating an abnormality (abnormal data) and stores it in the inspection work database 160. A sample indicating an abnormality is an abnormal sound of equipment or a device, an abnormal sound in an impact sound inspection, the appearance of a part where an abnormality has been found, etc. The inspection work support unit 116 instructs the user to express the sample indicating an abnormality in words. The inspection work support unit 116 executes data output processing (see FIG. 4) to obtain abnormal data desired by the user and stores it in the inspection work database 160.

[0065] ≪Fifth Embodiment: Features of Data Output Device≫ According to the data output device 100D, data of sounds and appearances indicating abnormalities are stored in the inspection work database 160 for equipment, devices, and their parts where no abnormalities were found during the inspection work. It becomes possible to convey inspection know-how to inexperienced workers with little experience and who have never encountered an abnormality. Note that in the inspection work support process (see FIG. 12), although abnormal data is generated during the inspection work, it is not limited to this. The data output device 100D may generate abnormal data of the inspection work target designated by the user after the inspection work is completed.

[0066] <<Fifth Embodiment: Variation: Data Stability>> By using the data output device 100D, abnormal data from multiple users (skilled workers) will be stored in the inspection work database 160. The data output device 100D may use the difference in abnormal data for the same abnormality to evaluate the quality or stability of the abnormal data as a sample. For example, the inspection work support unit 116 may determine that the smaller the variance of the generated abnormal data, the higher the quality or stability, or that the larger the number of abnormal data, the higher the quality or stability.

[0067] As described above, the inspection work support unit 116 calculates the quality or stability based on the variance of abnormal data of multiple users.

[0068] <<Fifth Embodiment: Variation: Acquisition of Abnormal Data>> The inspection work support unit 116 may search the inspection work database 160 to obtain and output abnormal data for the inspection target specified by the user.

[0069] As described above, the inspection work support unit 116 outputs abnormal data related to the instructed object (for example, equipment, machinery, or a building / construction as described later).

[0070] <<Sixth Embodiment>> In the fifth embodiment, the data output device 100D generates data on sounds and appearances indicating abnormalities. Instead of generating data on the sounds and appearances themselves, related data may also be generated. For example, if the state where rust has occurred on the handle is considered abnormal, instead of generating an image of the rust and storing it in the inspection work database 160, an image of the handle with rust may be generated and stored.

[0071] FIG. 13 is a diagram for explaining a procedure for generating an image 754 of a rusty handle according to the sixth embodiment. When the user desires an image of a rusted handle, the user instructs a data output device 100E (see FIG. 14 described later) to generate an image 751 of rust. Next, the user takes a picture of the handle to obtain an image 752 (auxiliary data). Subsequently, the user designates a region 753 of the handle where rust occurs and instructs to synthesize it, thereby obtaining an image 754 of the rusty handle.

[0072] FIG. 14 is a functional block diagram of a data output device 100E according to the sixth embodiment. Compared with the data output device 100D (see FIG. 11), the inspection work support unit 116E is different, and a synthesis unit 117 is added to the control unit 110. The process of generating the rust image 751 is the same as the data output process (see FIG. 4) according to the first embodiment. The data output device 100E generates a rust image 751 desired by the user, for example, according to the first input information "red rust" and the evaluation result "black is mixed in some places". Note that "red rust" and "black is mixed in some places" are information input by the user as text using a keyboard or the like, or information (language information) uttered toward a microphone (see the evaluation results 612 and 614 shown in FIG. 3).

[0073] Subsequently, the inspection work support unit 116E acquires and displays the image 752 of the handle taken by the user, and further accepts the designation of the region 753 of the handle where rust occurs designated by the user. Next, the synthesis unit 117 pastes the rust image 751 on the handle region 753 to generate an image 754 (non-verbal information) of the rusty handle and stores it in the inspection work database 160.

[0074] ≪Sixth Embodiment: Features of Data Output Device≫ When an example of an abnormality is a rusty handle, it is considered that it is easier for a non-expert to understand by using an image of a handle with rust closer to the actual object than using just an image of rust. By using the data output device 100E, a user who is an expert can generate such data, making it more efficient to convey the know-how of inspection. Although an image has been used as an example here, data in other output formats (modalities) may also be used. Regarding abnormal sounds that sometimes or periodically mix with normal operating sounds (the surrounding environment), the data output device 100E may generate and acquire abnormal sounds and synthesize them with the recorded normal operating sounds.

[0075] <<Modification Example: Data Generation Using Auxiliary Data>> In the sixth embodiment, the data output device 100E synthesizes the rust image 751 and the handle image 752 (auxiliary data) to generate the handle image 754 with rust. The generation unit 113F may directly generate an image of a handle with rust using a data generation model 130F (see FIG. 15 described later) having a synthesis function.

[0076] FIG. 15 is a diagram for explaining the data generation model 130F according to a modification example of the sixth embodiment. In addition to the input information 761, which is the language expression to be generated, the input to the data generation model 130F includes auxiliary data corresponding to a part of the input information 761. In FIG. 15, the auxiliary data is the image 762 corresponding to the handle of the "rusty handle" which is the input information 761, and the region 763 corresponding to the rusty part. For such an input, the data generation model 130F generates an image 765 of a handle with rust in the region 763. Such a data generation model 130F can be configured by combining the data generation model 130 and a machine learning model corresponding to the processing of the synthesis unit 117. Also, the data generation model 130F can be configured by using the inpainting technique in the image generation AI technology.

[0077] As described above, the inspection work support unit 116 stores, as abnormal data related to the object, data (see image 754) in which the generated data (see image 751) is combined with at least one of auxiliary data indicating data of an object (see image 752) and data indicating the environment around the object (for example, the normal operating sound of equipment / devices).

[0078] ≪Modification Example: Specification of Output Format≫ In the above-described embodiments, the input to the data output devices 100, 100A, 100B, 100C, 100D, 100E is language information (for example, "a sound like clank") indicating an output that is non-verbal information (for example, a sound). As inputs other than the input information 611, 651, 761, output formats (modalities) such as sounds and smells may be included.

[0079] The output format is not limited to one, and may be plural. For example, for the input information "rusty handle", an image and a tactile sensation may be output. When a plurality of output formats are specified, the generation unit 113 may use a plurality of data generation models 130 that generate data of each output format to generate data of each output format, or may use a data generation model that generates data of a plurality of output formats (multi-modal).

[0080] ≪Other Modification Examples≫ Although some embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. For example, it is possible to similarly generate data of other modalities such as smells, tastes, tactile sensations, and vibrations, not limited to sound and visual sound data / images indicating abnormalities or signs of abnormalities in equipment and devices. Also, it can be used not only for sounds and images referred to during inspection, but also for generating sound and image data referred to during inspection before product shipment. The objects of inspection and examination are not limited to equipment and devices, and may include objects such as buildings / buildings including buildings, bridges, and tunnels.

[0081] The present invention can take various other embodiments, and furthermore, various changes such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and its equivalent scope.

[0082] ≪Hardware Configuration≫ The data output devices 100, 100A, 100B, 100C, 100D, 100E according to the above-described embodiments are realized by a computer 900 having a configuration as shown in FIG. 16, for example. FIG. 16 is a hardware configuration diagram showing an example of a computer 900 that realizes the functions of the data output devices 100, 100A, 100B, 100C, 100D, 100E according to the above-described embodiments. The computer 900 includes a CPU 901, a ROM 902, a RAM 903, an SSD 904, an input / output interface 905 (described as input / output I / F (Interface) in FIG. 16), a communication interface 906 (described as communication I / F in FIG. 16), and a media interface 907 (described as media I / F in FIG. 16). The computer 900 may include an HDD (Hard Disc Drive) instead of the SSD 904, or may further include an HDD in addition to the SSD 904.

[0083] The CPU 901 operates based on a program stored in the ROM 902 or the SSD 904, and performs control by the control unit 110 in FIG. 2. The ROM 902 stores a boot program executed by the CPU 901 when the computer 900 is started up, a program related to the hardware of the computer 900, and the like. The CPU 901 controls an input device 910 such as a mouse and a keyboard, and an output device 911 such as a display and a printer via the input / output interface 905. The CPU 901 acquires data from the input device 910 via the input / output interface 905, and outputs the generated data to the output device 911.

[0084] The SSD904 stores programs executed by the CPU901, data used by such programs, and the like. The communication interface 906 receives data from another device (not shown) via a communication network, outputs the data to the CPU901, and transmits data generated by the CPU901 to another device via the communication network. The media interface 907 reads a program or data stored in the recording medium 912 and outputs the program or data to the CPU901 via the RAM903. The CPU901 loads a program from the recording medium 912 onto the RAM903 via the media interface 907 and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical disk), a magnetic recording medium, a conductor memory tape medium, or a semiconductor memory.

[0085] For example, when the computer 900 functions as the data output devices 100, 100A, 100B, 100C, 100D, 100E according to the above-described embodiment, the CPU901 of the computer 900 realizes the functions of the data output devices 100, 100A, 100B, 100C, 100D, 100E by executing the program 128 (see FIG. 2) loaded onto the RAM903. The CPU901 reads and executes a program from the recording medium 912. Alternatively, the CPU901 may read a program from another device via a communication network, or may install and execute the program 128 from the recording medium 912 to the SSD904.

Explanation of Reference Numerals

[0086] 100, 100A, 100B, 100C, 100D, 100E Data output device 111 Input acquisition unit 112 Attribute acquisition unit 113, 113F Generation unit 114 Evaluation result acquisition unit 115, 115A, 115B, 115C Update unit 116, 116E Inspection work support unit 117 Synthesis Department 121 Sample Data 130, 130A, 130F Data Generation Model (Machine Learning Model) 140 User Attribute Database 150 User Information Database 160 Inspection Work Database 191 Input Device (Input Unit) 192 Output Device (Output Unit) 611, 613, 651, 653, 712, 714, 716, 761 Input Information 612, 614, 652, 654, 711, 713, 715 Evaluation Results 621, 622 Parameters (User Attributes) 631, 632, 633, 671, 672, 673, 731 - 736 Sound Data (Data) 661, 662 Latent Variables (User Attributes) 751, 765 Images (Data)

Claims

1. An input unit that acquires input information input as information indicated by a language by a user; A generation unit that generates, based on a user attribute that is an attribute of the user, data of non-verbal information from the input information acquired by the input unit and corresponding to the input information indicated by the language; An output unit that outputs the generated data so as to be perceivable by at least one of the five human senses, and comprising A data output device.

2. An evaluation result acquisition unit that acquires an evaluation result of a user with respect to an output result output by the output unit; An update unit that repeatedly instructs the generation unit to generate data until an evaluation result that the evaluation result of the output result of the data corresponding to the input information reflecting the evaluation result is the desired output result of the user; and comprising The data output device according to claim 1.

3. The update unit Updates the user attribute so that the data corresponding to the input information acquired by the input unit becomes data corresponding to the desired output result The data output device according to claim 2.

4. The generation unit Generates data corresponding to the input information by using a learned machine learning model that uses the input information indicated by the language as an explanatory variable and outputs data corresponding to the input information; The user attribute is a parameter of the machine learning model The data output device according to claim 3.

5. The generation unit Generates data corresponding to the input information by using a learned machine learning model that uses the input information indicated by the language and a latent variable or a distribution of latent variables indicating the user attribute as explanatory variables and outputs data corresponding to the input information The data output device according to claim 3.

6. The evaluation result acquisition unit Acquires information indicated by a language corresponding to an output result of sample data; The update unit Updates the user attribute so that the data corresponding to the information becomes the sample data The data output device according to claim 3.

7. The update unit Calculates the user attribute of a new user based on the user attributes of a user who has similar user information to the user information of the new user and who has an updated user attribute The data output device according to claim 3.

8. The input information indicates an abnormality of an object or a sign of an abnormality; A inspection work support unit that stores the generated data as abnormality data related to the object The data output device according to claim 2.

9. The inspection work support unit stores, as abnormal data related to the object, data obtained by synthesizing the generated data with auxiliary data that is at least one of the data indicating the object and the data indicating the environment around the object. The data output device according to claim 8.

10. The inspection work support unit calculates the quality or stability based on the dispersion of the abnormal data of a plurality of users. The data output device according to claim 8.

11. The inspection work support unit outputs the abnormal data related to the instructed object. The data output device according to claim 8.

12. A data output device performs the steps of: obtaining input information input as information indicated by a language by a user; generating, based on user attributes that are the attributes of the user, the data corresponding to the input information indicated by the language as non-verbal information data; and outputting the generated data so as to be perceivable by at least one of the five human senses. A data output method. ​

Images