Estimation method and estimation device

Abstract

This estimation method estimates the work of a worker by a computer, and the computer acquires work sound data attached to the collected work (S01), and estimates whether or not the worker is performing a work for processing a transparent object by inputting the work sound data into a learned first model (S02).

Description

Technical Field

[0001] The present disclosure relates to an estimation method and the like for estimating an operator's work. Background Art

[0002] As the first stage for improving the productivity of a factory, data related to the work performed by the operator is automatically collected to classify the work, and the time required for each classification is measured. For example, Patent Document 1 discloses a technology for classifying the work by recognizing the objects (such as transparent objects) handled in the work based on images captured under multiple camera conditions.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-017653 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, in the technology described in Patent Document 1, even if the imaging conditions are changed, when the transparency of the object is high or when the change in the refractive index or reflectivity of the light of the object is small, the recognition accuracy of the object is low. Therefore, in the technology described in Patent Document 1, there is a case where the operation of processing an object with high transparency (hereinafter referred to as a transparent object) cannot be estimated with good accuracy.

[0008] Therefore, the present disclosure provides an estimation method and the like that can estimate the operation of handling a transparent object with high accuracy.

[0009] Means used to solve problems

[0010] An inference method of a technical solution related to the present invention is a method for inferring an operator's work performed by a computer, wherein the computer obtains data of work sound associated with the work that is collected; the computer infers whether the operator is performing an operation of handling a transparent object by inputting the work sound data into a learned first model.

[0011] Effects of the Invention

[0012] According to the present disclosure, it is possible to estimate the operation of handling a transparent object with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a block diagram showing an example of the functional configuration of the estimation system according to the embodiment.

[0014] Figure 2 This is a flowchart showing an operation example 1 of the estimation system according to the embodiment.

[0015] Figure 3 It indicates schematically Figure 2 A diagram showing an example of the process in step S02.

[0016] Figure 4 This is a graph showing the similarity between the feature amount of the collected working sounds and the feature amount of the working sounds of the work of handling a transparent object.

[0017] Figure 5 This is a diagram showing the result of analyzing the working sound for one hour in time series in Verification Example 1.

[0018] Figure 6 This is a diagram for explaining the estimation method of the bag operation executed in Verification Example 3.

[0019] Figure 7 This is a diagram showing an example of the architecture of a neural network.

[0020] Figure 8 This is a diagram showing a method of calculating the correct answer rate when two classifications are estimated.

[0021] Fig. 9 This is a graph showing the inference results of two categories and the correct answer rate in Verification Example 3.

[0022] Fig.10 This is a diagram showing a method of calculating the correct answer rate when three classifications are estimated.

[0023] Fig.11 This is a graph showing the inference results of the three categories of Verification Example 3 and the correct answer rates.

[0024] Fig.12 This is a diagram showing a method of estimating two categories based on a combination of input data and a method of calculating the correct answer rate.

[0025] Fig.13 This is a diagram showing the estimation results of two classifications based on the combination of input data and the correct answer rate in Verification Example 3.

[0026] Fig.14 This is a diagram showing the comparison results of the estimation accuracy between the estimation method using image AI and the estimation method of Operation Example 1.

[0027] Fig.15 This is a diagram for explaining the difference between the estimation result using the data of the working sound and the estimation result using the data of the image.

[0028] Fig.16 This is a diagram for explaining an outline of the flow of operation example 2 of the estimation system according to the embodiment.

[0029] Fig.17 This is a flowchart showing an operation example 2 of the estimation system according to the embodiment.

[0030] Fig.18 This is a flowchart showing Modification 1 of Operation Example 2 of the estimation system according to the embodiment.

[0031] Fig.19 This is a diagram schematically showing a configuration example 1 of an estimation unit that executes a flow of a modification example 1 of an operation example 2.

[0032] Fig. 20 This is a diagram for explaining the estimation method of the bag operation executed by the configuration example 1.

[0033] Fig.21 This is a diagram schematically showing a configuration example 2 of an estimation unit that executes a flow of a modification example 1 of an operation example 2.

[0034] Fig. 22 This is a diagram schematically showing a configuration example 3 of an estimation unit that executes a flow of a modification example 1 of the operation example 2.

[0035] Fig.23 This is a diagram showing an example of the architecture of the image sub-network.

[0036] Fig.24 This is a diagram showing an example of the architecture of a sound subnetwork.

[0037] Fig.25 This is a diagram showing an example of the architecture of the fusion layer.

[0038] Fig.26 is a diagram showing an example of the architecture of a classification network.

[0039] Fig. 27 This is a diagram showing an example of the architecture of a contrastive learning network.

[0040] Fig.28 This is a diagram schematically showing a configuration example of an estimation unit for executing a process of a variation 2 of the operation example 2.

[0041] Fig.29 This is a diagram showing an example of a work sound in which the estimation unit mistakenly estimates that the worker is performing a work of handling a transparent object.

[0042] Fig. 30A This is a flowchart of operation example 3 of the estimation system according to the embodiment.

[0043] Fig. 30B This is a flowchart showing an example of an operation of preregistering feature amounts of working sounds that may be erroneously estimated.

[0044] Fig.31This is a block diagram showing an example of the functional configuration of an estimation system according to another embodiment. DETAILED DESCRIPTION

[0045] (Insights that form the basis of this disclosure)

[0046] As the first step to improve factory productivity, data related to the work performed by operators is automatically collected and the work is classified, and the time required for each classification is measured. In this way, users can understand which work the operator needs time for, so they can set up work plans so that the operator can perform the work more efficiently.

[0047] In the past, the status of the work performed by the operator was photographed with a camera, and the work was classified by identifying the object handled by the operator. For example, in Patent Document 1, a transparent object is identified based on a plurality of images taken by changing the camera conditions, and the work is classified as the work that the operator is doing to handle the transparent object. However, for an object with high transparency (i.e., a transparent object), even if the camera conditions are changed, it is difficult to identify the object through the image when the transparency is high or the change in the refractive index or reflectivity of the light of the object is small. Therefore, in the technology described in Patent Document 1, there is a situation where the work of handling the transparent object cannot be accurately estimated.

[0048] Therefore, the following method is required: by accurately identifying the object being processed during the operation, even when the transparency of the object being processed is high or the change in the refractive index or reflectivity of the light of the object is small, and accurately estimating the operation of processing the transparent object, the operator's operation accuracy can be classified.

[0049] In addition, in the conventional methods, the focus is mainly on identifying the object by photographing the stationary object with a camera. In this regard, the inventors of the present application have found that by picking up the operation sound accompanying the operation (in other words, the sound generated along with the operation), even if the transparent object moves or deforms during the operation performed by the operator, the operation of handling the transparent object can be accurately estimated.

[0050] (Overview of the present disclosure)

[0051] An inference method in Example 1 of a technical solution disclosed herein is a method for inferring an operation of an operator performed by a computer, wherein the computer obtains data of work sounds collected during the operation; the computer infers whether the operator is performing an operation of handling a transparent object by inputting the data of the work sounds into a first model that has been learned.

[0052] Thus, the device that executes the estimation method uses the first model that takes the operation sound data as input and outputs whether the operation is to handle a transparent object, so it is possible to estimate the operation to handle a transparent object with good accuracy.

[0053] In addition, in the inference method of Example 2 of a technical solution of the present disclosure, in the inference method of Example 1, the computer obtains data of an image of the operator performing the operation corresponding to the data of the operation sound; the computer infers whether the operator is performing the operation of processing the transparent object by inputting the data of the image into the learned second model; the computer infers whether the operator is performing the operation of processing the transparent object based on the inference result using the first model and the inference result using the second model. In addition, the inference result using the first model is the inference result inferred by the first model based on the data of the operation sound, and the inference result using the second model is the inference result inferred by the second model based on the data of the image.

[0054] Thus, the device executing the estimation method estimates whether the worker is performing an operation of handling a transparent object based on the estimation result estimated by the first model based on the data of the operation sound and the estimation result estimated by the second model based on the data of the image. Therefore, the device executing the estimation method can estimate the operation of handling a transparent object with higher accuracy than when estimating using only the data of the operation sound.

[0055] In addition, in the inference method of Example 3 of a technical solution disclosed in the present invention, in the inference method of Example 1, the computer may obtain data of an image of the worker performing the operation corresponding to the data of the operation sound; and the computer may infer whether the worker is performing the operation of handling the transparent object by inputting the data of the operation sound and the data of the image into the first model.

[0056] Therefore, the device that executes the inference method uses the data of the work sound and the data of the image corresponding to the work sound as input and outputs the first model of whether the work is to process a transparent object. Therefore, compared with the case where only the data of the work sound is used for inference, it can more accurately infer the work of processing a transparent object.

[0057] In addition, in the inference method of Example 4 of a technical solution related to the present disclosure, in the inference method of any one of Examples 1 to 3, the computer may infer whether the operator is performing the operation of processing the transparent object based on the similarity between the feature value of the operation sound output from the first model and the feature value of the operation sound of the operation of processing the transparent object pre-stored in the storage unit.

[0058] Therefore, the device that executes the inference method can infer whether the worker is performing the operation of processing a transparent object based on the similarity between the characteristic quantity of the working sound output from the first model and the characteristic quantity of the working sound of the operation of processing a transparent object, so it can infer the operation of processing a transparent object with good accuracy.

[0059] In addition, in the inference method of Example 5 of a technical solution related to the present disclosure, in the inference method of any one of Examples 1 to 4, the computer may further infer whether the operator is performing the operation of processing the transparent object based on the similarity of the feature quantity of the work sound output from the first model to the feature quantity of the work sound of the operation of processing the transparent object pre-stored in the storage unit, and the similarity to the feature quantity of the work sound pre-stored in the storage unit that may be mistakenly inferred that the operator is performing the operation of processing the transparent object.

[0060] Thus, the device executing the estimation method can reduce the occurrence of mis-estimation by comparing the similarity between the feature quantity of the operation sound output from the first model and the feature quantity of the operation sound of the operation of handling the transparent object and the feature quantity of the operation sound output from the first model that may be mis-estimation as the feature quantity of the operation sound. Therefore, the device executing the estimation method can estimate the operation of handling the transparent object with good accuracy even if only the data of the operation sound is used.

[0061] In addition, in the inference method of Example 6 of a technical solution disclosed in the present invention, in the inference method of Example 5, the computer infers that the operator is performing the operation of processing the transparent object when the similarity between the feature quantity of the work sound output from the first model and the feature quantity of the work sound of the operation of processing the transparent object is higher than the similarity between the feature quantity of the work sound that may be mistakenly inferred that the operator is performing the operation of processing the transparent object.

[0062] As a result, the device that executes the estimation method can reduce the occurrence of erroneous estimation, so even if only the data of the operation sound is used, the operation of handling the transparent object can be estimated with high accuracy.

[0063] In addition, in the inference method of Example 7 of a technical solution disclosed in the present invention, in the inference method of Example 5 or Example 6, when the computer inputs data of the work sound of the work process of processing a non-transparent object different from the transparent object into the first model and the similarity between the feature quantity of the work sound of the work process of processing the non-transparent object and the feature quantity of the work sound of the work process of processing the transparent object is higher than a threshold value, the computer determines the work sound of the work process of processing the non-transparent object as a work sound that may be mis-inferred, and the work sound that may be mis-inferred is a work sound that may be mis-inferred as the work sound of the work process of processing the transparent object; the computer saves the feature quantity of the work sound of the work process of processing the non-transparent object as the feature quantity of the work sound that may be mis-inferred in the storage unit.

[0064] Thus, the device that executes the inference method can accurately determine whether the working sound of the operation of processing the non-transparent object is a working sound that may be mistakenly inferred as the operation of processing the transparent object based on the similarity between the characteristic amount of the working sound of the operation of processing the non-transparent object and the characteristic amount of the working sound of the operation of processing the transparent object. Therefore, the device that executes the inference method can save the characteristic amount of the working sound that is more likely to be misjudged in the storage unit. Therefore, the device that executes the inference method can reduce the occurrence of misjudgment by using the characteristic amount of the working sound that may be misjudged stored in the storage unit, so even if only the data of the working sound is used, the operation of processing the transparent object can be accurately inferred.

[0065] In an inference method of Example 8 according to one aspect of the present disclosure, in the inference method of any one of Examples 1 to 7, the data of the work sound may include data of a sound in an inaudible frequency band.

[0066] Thus, the device that executes the inference method uses the data of the working sound including the sound from the audible frequency band to the sound from the inaudible frequency band to infer whether the operator is performing the operation of handling the transparent object. In this way, by including the sound from the inaudible frequency band in the working sound data, the noise from the environment that becomes the cause of the mis-inference in the working sound data is reduced, so the device that executes the inference method can improve the inference accuracy of the operation of handling the transparent object. Furthermore, the device that executes the inference method can infer whether the operator is performing the operation of handling the transparent object based on more information than the case of using only the data of the sound from the audible frequency band. Therefore, the device that executes the inference method can infer the operation of handling the transparent object with better accuracy.

[0067] In addition, an inference device of Example 9 of a technical solution related to the present disclosure is an inference device for inferring an operator's work, comprising: an acquisition unit, which acquires the collected data of work sound associated with the work; and an inference unit, which infers whether the operator is performing an operation of handling a transparent object by inputting the data of the work sound into a first model that has been learned.

[0068] Thus, the estimation device uses the first model that takes the data of the operation sound as input and outputs whether the operation is to handle a transparent object, and thus can estimate the operation to handle a transparent object with high accuracy.

[0069] Furthermore, a program according to Example 10 of one technical aspect of the present disclosure is a program for causing a computer to execute the estimation method according to any one of Examples 1 to 8.

[0070] In this way, it is possible to achieve the same effect as the above-mentioned estimation method using a computer.

[0071] In addition, these inclusive or specific technical solutions can also be implemented by systems, methods, devices, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs (Compact Disc Read Only memory), or by any combination of systems, methods, devices, integrated circuits, computer programs, and recording media.

[0072] Below, while referring to the attached Figure 1 The embodiments of the present disclosure are specifically described. The numerical values, shapes, materials, constituent elements, configuration positions of constituent elements and connection forms, steps, order of steps, etc. shown in the following embodiments are examples and are not intended to limit the scope of the claims. In addition, regarding the constituent elements of the following embodiments that are not recorded in the independent claims representing the highest concept, they are described as arbitrary constituent elements. In addition, the figures are not necessarily illustrated strictly. In each figure, the same reference numerals are given to substantially the same structure, and repeated descriptions may be omitted or simplified.

[0073] In the present disclosure, terms such as parallel and perpendicular indicating the relationship between elements and terms and numerical values ​​such as rectangle indicating the shape of an element do not have strict meanings but mean substantially the same range and include a difference of several percentage points, for example.

[0074] (Implementation Method)

[0075] Below, while referring to the attached Figure 1 The implementation modes will be described in detail.

[0076] [1. Summary]

[0077] First, an overview of the estimation system according to the embodiment will be described. Figure 1 This is a block diagram showing an example of the functional configuration of the estimation system 200 according to the embodiment.

[0078] The estimation system 200 is a system for estimating the operation of the operator. The estimation system 200 obtains the operation sound accompanying the operation collected by the sound pickup device 10, for example, and inputs the data of the operation sound into the learned first model 132 (hereinafter also referred to as the first model 132) to estimate whether the operator is performing the operation of handling a transparent object.

[0079] The estimation system 200 can display the estimation result estimated by the estimation device 100 on the display unit of the information terminal 50 and prompt the user. Thus, the user can refer to the estimation result to understand the time required for the operation of processing transparent objects and the operation of processing non-transparent objects. In addition, the user can refer to the estimation result to set up the operation plan of the operator, so the efficiency of the operation in the work space 80 can be improved.

[0080] The operation sound accompanying the operation includes the sound generated along with the operation. The operation sound is, for example, the sound generated when the object handled by the operator is moved or deformed. The operation is, for example, picking up parts, cleaning operations, inspection or packaging. The operation space 80 refers to, for example, a space where the operator performs the operation in a manufacturing plant or a logistics warehouse.

[0081] In addition, a transparent object is an object with high transparency, for example, it is made of a material with high transparency such as synthetic resin or glass. High transparency means that, for example, when the object is in the form of a sheet or is made of a sheet, the haze of the sheet is less than 0.5%, and when the object is in the form of a flat plate or a block or is made of a flat plate or a block, the refractive index of light is greater than 1.30 and less than 1.70. Transparent objects are, for example, containers, bags, cushioning materials, or parts.

[0082] As synthetic resins, for example, vinyl resins such as polyvinyl chloride resins, polycarbonate resins, polyester resins, polyethylene naphthalate resins, polyethylene resins, polypropylene resins, polyimide resins, polystyrene resins, polyurethane resins, acrylic resins, fluororesins, etc. In addition, the materials constituting the highly transparent object are not limited to the above examples, and for example, natural polymers such as fine fibrous cellulose may also be included.

[0083] In addition, the inference system 200 can obtain the data of the image showing the worker performing the operation captured by the camera device 20, and input the obtained image data and the operation sound data into the first model 132 to infer whether the worker is performing the operation of handling the transparent object, or can infer whether the worker is performing the operation of handling the transparent object based on the inference result obtained by inputting the image data into the learned second model 133 (hereinafter, also referred to as the second model 133) and the inference result obtained by inputting the operation sound data into the first model 132. The image data corresponds to the operation sound data.

[0084] [2. Structure]

[0085] Next, refer to Figure 1 The structure of the estimation system 200 of the embodiment is described. The estimation system 200 includes, for example, a sound pickup device 10, a camera device 20, an information terminal 50, and an estimation device 100. The sound pickup device 10 and the camera device 20 are installed in a space (working space 80) where the operator performs work, and are connected to the information terminal 50 and the estimation device 100 via communication. Figure 1 The configuration of the estimation system 200 shown is just an example, and the system is not limited to this example.

[0086] [Sound pickup device 10]

[0087] The sound pickup device 10, for example, collects the work sounds incidental to the work of the worker. The sound pickup device 10 is, for example, set in the work space 80. The sound pickup device 10 can collect sounds from the audible frequency band to the inaudible frequency band. The audible frequency band is a frequency band that can be perceived by the human ear, and the inaudible frequency band is a frequency band that cannot be perceived by the human ear. The sound in the inaudible frequency band is, for example, the sound in the frequency band above 20kHz. More specifically, the sound pickup device 10 is a microphone, for example, it can be a (Micro Electro Mechanical Systems) microphone or a laser microphone.

[0088] The sound pickup device 10, for example, can collect sounds with a wider frequency band than a normal microphone when it is a laser microphone. In addition, since a laser microphone does not have a vibration plate like a normal microphone, it can also collect sounds in an environment such as electromagnetic waves, high temperature or high heat.

[0089] exist Figure 1 , an example is shown in which the inference system 200 has one sound pickup device 10, but it may also have two or more sound pickup devices 10. In addition, the sound pickup device 10 may also be a directional microphone. Thus, the sound pickup device 10 is less likely to collect sounds that become noises such as surrounding noises, so it is possible to collect working sounds with high sensitivity.

[0090] The sound collection device 10 converts the collected sound (working sound) into an electric signal and outputs it to the estimation device 100. Alternatively, the sound collection device 10 may add a time stamp and its own identification number to the collected working sound data and output it to the estimation device 100.

[0091] [Camera 20]

[0092] The camera device 20, for example, captures an image showing a worker performing a work. The data of the image corresponds to the data of the work sound collected by the sound pickup device 10. That is, the camera device 20 may also operate in conjunction with the sound pickup device 10, for example, by assigning a timestamp to the acquired data (the data of the work sound and the data of the image), thereby establishing a correspondence between the data of the work sound and the data of the image. At this time, for example, the camera device 20 may also assign its own identification number to the image data. The camera device 20 is, for example, set in the work space 80. The camera device 20 is, for example, an RGB camera, but may also include distance data.

[0093] The imaging device 20 outputs the data of the captured image to the estimation device 100 .

[0094] [Information terminal 50]

[0095] The information terminal 50 is an information terminal used by the user, such as a personal computer or a tablet terminal. The information terminal 50 displays the inference result inferred by the inference device 100 on the display unit. In addition, the information terminal 50 receives an instruction input by the user and sends the instruction to the sound collection device 10, the camera device 20 and the inference device 100.

[0096] [Estimation device 100]

[0097] The estimation device 100 is a device for estimating the operation of the operator. For example, the estimation device 100 acquires data of operation sound accompanying the operation collected by the sound collection device 10 and inputs the data of the operation sound into the learned first model 132 to estimate whether the operator is performing an operation of handling a transparent object.

[0098] For example, Figure 1 As shown in FIG. 1 , the inference device 100 includes a communication unit 110, an information processing unit 120, a storage unit 130, a model generation unit 140, and an input receiving unit 150. The inference device 100 is, for example, a server device. Figure 1 In the example of , the estimation device 100 includes the second model 133, but does not necessarily need to include the second model 133. Hereinafter, each structure of the estimation device 100 will be described.

[0099] [Communication Department 110]

[0100] The communication unit 110 is a communication circuit (communication module) for the inference device 100 to communicate with the sound pickup device 10 and the camera device 20. The communication unit 110 has a communication circuit (communication module) for communicating via a wide area communication network, but may also have a communication circuit (communication module) for communicating via a local communication network. The communication unit 110 is, for example, a wireless communication circuit for wireless communication, but may also be a wired communication circuit for wired communication. In addition, the communication standard of the communication performed by the communication unit 110 is not particularly limited.

[0101] [Information processing unit 120]

[0102] The information processing unit 120 performs various information processing related to the inference device 100. More specifically, for example, the information processing unit 120 obtains data of the working sound collected by the sound pickup device 10 (for example, an electrical signal of the working sound), and performs various information processing related to the inference of whether the worker is performing the work of processing the transparent object. In addition, for example, the information processing unit 120 may also obtain data of an image showing the worker performing the work captured by the camera device 20, and perform various information processing related to the inference of whether the worker is performing the work of processing the transparent object. The information processing unit 120 may use the data of the working sound to infer the work, or use the data of the working sound and the data of the image to infer the work. Specifically, the information processing unit 120 includes an acquisition unit 121 and an inference unit 122. The functions of the acquisition unit 121 and the inference unit 122 are realized by the processor or microcomputer constituting the information processing unit 120 executing a computer program stored in the storage unit 130.

[0103] [Acquisition section 121]

[0104] The acquisition unit 121 acquires, for example, data of work sounds collected by the sound collection device 10. The work sound data is the sound accompanying the work of the worker, for example, the sound generated along with the work performed by the worker. In addition, the acquisition unit 121 acquires, for example, data of an image of the worker performing the work, which is captured by the camera device 20 and corresponds to the work sound data. The work sound data may be an image of a frequency spectrum created by Fourier transforming the electrical signal of the work sound collected by the sound collection device 10, or may be numerical data of a time series.

[0105] [Estimation Unit 122]

[0106] If the acquisition unit 121 acquires the data of the operation sound, the estimation unit 122 estimates whether the operator is performing the operation of handling a transparent object based on the data of the operation sound. The estimation unit 122 estimates whether the operator is performing the operation of handling a transparent object by, for example, inputting the data of the operation sound to the first model 132 that has been learned (hereinafter referred to as the first model 132). Specifically, for example, the estimation unit 122 estimates whether the operator is performing the operation of handling a transparent object based on the similarity between the feature amount of the operation sound output from the first model 132 and the feature amount of the operation sound of the operation of handling a transparent object pre-stored in the storage unit 130 (for example, the feature amount database 131 in the storage unit 130). More specifically, for example, the inference unit 122 may input the data of the operation sound into the first model 132, calculate the similarity between the feature quantity of the operation sound of the operation of handling transparent objects extracted by the first model 132 and the feature quantity of the operation sound of the operation of handling transparent objects pre-stored in the storage unit 130, and in the case where the calculated similarity is greater than a predetermined value (i.e., a threshold value), infer that the worker is performing the operation of handling transparent objects. In addition, without being limited to this example, the inference unit 122 may also use a model that directly outputs the inference result of whether the worker is performing the operation of handling transparent objects based on the data of the operation sound.

[0107] Furthermore, if the acquisition unit 121 acquires data of an image showing a worker performing a work corresponding to the data of the work sound, the estimation unit 122 may also estimate whether the worker is performing a work of handling a transparent object based on the data of the work sound and the data of the image. Specifically, for example, the estimation unit 122 estimates whether the worker is performing a work of handling a transparent object by inputting the data of the work sound and the data of the image showing a worker performing a work corresponding to the data of the work sound into the first model 132. The details of the first model 132 will be described later.

[0108] Furthermore, for example, when the estimation device 100 has the second model 133 that has been learned, if the acquisition unit 121 acquires the above-mentioned image data, the estimation unit 122 estimates whether the worker is performing the operation of handling a transparent object by inputting the image data into the second model 133. At this time, the estimation unit 122 estimates whether the worker is performing the operation of handling a transparent object by inputting the data of the operation sound of the worker performed in the image data acquired by the acquisition unit 121 into the first model 132. Furthermore, the estimation unit 122 estimates whether the worker is performing the operation of handling a transparent object based on the estimation result estimated from the image data using the second model 133 and the estimation result estimated from the operation sound data using the first model 132.

[0109] In addition, the estimation unit 122 can determine, for example, whether the working sound collected by the sound collection device 10 is a working sound that may be mistakenly estimated as the working sound of the working process of processing a transparent object. Specifically, for example, when the similarity between the feature quantity of the working sound of the working process of processing a non-transparent object obtained by inputting the data of the working sound of the working process of processing a non-transparent object different from the transparent object into the first model 132 and the feature quantity of the working sound of the working process of processing a transparent object is higher than a predetermined value (i.e., a threshold value), the estimation unit 122 determines that the working sound of the working process of processing a non-transparent object is a working sound that may be mistakenly estimated by the estimation unit 122 as the working sound of the working process of processing a transparent object. And, the estimation unit 122 stores the feature quantity of the working sound determined as the working sound that may be mistakenly estimated in the feature quantity database 131 (feature quantity DB) of the storage unit 130.

[0110] In addition, feature quantities of working sounds of working operations for handling transparent objects stored in advance may be stored in the feature quantity database 131. The feature quantity database 131 will be described later.

[0111] [Storage unit 130]

[0112] The storage unit 130 is a storage device that stores dedicated application programs and the like for the information processing unit 120 to perform various information processing. For example, the storage unit 130 stores a feature database 131, a first model 132, and a second model 133. The storage unit 130 is implemented by, for example, a HDD (Hard Disk Drive), but may also be implemented by a semiconductor memory.

[0113] The feature database 131 stores the feature of the work sound extracted in advance. The feature can be expressed by a numerical value or a combination of numerical values ​​as an embedding (such as a tensor, a matrix, etc.), an embedding vector, or a variance expression. For example, the feature database 131 can store the feature of the work sound associated with the work of processing transparent objects and the feature of the work sound that may be mistakenly inferred as the work of the worker processing the transparent object. In addition, the feature database 131 can store the feature of the image extracted in advance. For example, the feature database 131 can store the feature of an image showing a worker performing the work of processing transparent objects (specifically, the feature of the transparent object appearing in the image).

[0114] The first model 132 is, for example, a learned model generated by the model generation unit 140. The first model 132, for example, takes the data of the operation sound as input, and outputs whether the worker is performing the operation of processing a transparent object. More specifically, the first model 132, for example, extracts the feature quantity of the input operation sound data, calculates the similarity between the extracted feature quantity and the feature quantity of the operation sound of the operation of processing a transparent object pre-stored in the storage unit 130, and in the case where the calculated similarity is greater than a predetermined value, it is inferred that the worker is performing the operation of processing a transparent object. In addition, for example, the first model 132 can also take the data of the image corresponding to the data of the operation sound, in which the worker performing the operation is shown, as input, and output whether the worker is performing the operation of processing a transparent object. More specifically, the first model 132 can, for example, extract the feature quantity of the input image data, calculate the similarity between the extracted feature quantity and the feature quantity of the image of the worker performing the operation pre-stored in the storage unit 130, and in the case where the calculated similarity is greater than a predetermined value, it is inferred that the worker is performing the operation of processing a transparent object.

[0115] The second model 133 is a learned model generated by the model generation unit 140. The second model 133, for example, takes as input data of an image showing a worker performing a work corresponding to the work sound data, and outputs whether the worker is performing a work of handling a transparent object. More specifically, the second model 133 can, for example, extract feature quantities of the input image data, calculate the similarity between the extracted feature quantities and feature quantities of an image showing a worker performing a work of handling a transparent object stored in advance in the storage unit 130, and infer that the worker is performing a work of handling a transparent object when the calculated similarity is greater than a predetermined value.

[0116] Alternatively, the first model 132 and the second model 133 may extract feature quantities from input data and output the extracted feature quantities.

[0117] Specifically, the first model 132 and the second model 133 are neural network models, such as convolutional neural network (CNN), recurrent neural network (RNN), or LSTM (Long-Short Term Memory).

[0118] [Model generation unit 140]

[0119] The model generation unit 140 generates the first model 132 and the second model 133 by, for example, performing machine learning using teacher data. For example, the model generation unit 140 generates, by machine learning, a sound recognition model (hereinafter also referred to as an audio subnetwork) that takes the data of the work sound as input and outputs whether the worker is performing the work of processing a transparent object. In addition, for example, the model generation unit 140 can further generate, by machine learning, an image recognition model (hereinafter also referred to as an image subnetwork) that takes the data of the image of the worker performing the work corresponding to the data of the work sound as input and outputs whether the worker is performing the work of processing a transparent object. The first model 132 can be, for example, a sound recognition model or a model including a sound recognition model and an image recognition model. The data of the work sound input to the first model 132 can be, for example, an image of a frequency spectrum or numerical data of a time series. The data of the work sound can include data of sound in a non-audible frequency band.

[0120] Furthermore, the model generation unit 140 may generate, by machine learning, an image recognition model (for example, the second model 133 ) that takes image data as input and outputs a feature quantity representing a transparent object appearing in the image.

[0121] As described above, the voice recognition model extracts the feature quantity of the input operation sound data, calculates the similarity between the extracted feature quantity and the feature quantity of the operation sound of the operation of handling transparent objects pre-stored in the storage unit 130, and infers that the worker is performing transparent operation when the calculated similarity is greater than a predetermined value. In addition, the image recognition model extracts the feature quantity of the input image data, calculates the similarity between the extracted feature quantity and the feature quantity of the image of the worker performing the operation of handling transparent objects pre-stored in the storage unit 130, and infers that the worker is performing the operation of handling transparent objects when the calculated similarity is greater than a predetermined value. In addition, the model including the voice recognition model and the image recognition model infers whether the worker is performing the operation of handling transparent objects based on the inference results using the two models.

[0122] The model generation unit 140 may update the first model 132 and the second model 133 by storing the learned model in the storage unit 130. The model generation unit 140 is implemented by, for example, a processor executing a program stored in the storage unit 130.

[0123] In addition, the first model 132 and the second model 133 can extract feature quantities of input data and output the extracted feature quantities.

[0124] [Input Accepting Unit 150]

[0125] The input accepting unit 150 is an input interface that receives operation inputs made by a user using the inference device 100. The input accepting unit 150 is specifically implemented by a touch panel display or the like. For example, when the input accepting unit 150 is equipped with a touch panel display, the touch panel display functions as a display unit (not shown) and the input accepting unit 150. In addition, the input accepting unit 150 is not limited to a touch panel display, and may be, for example, a keyboard, a pointing device (e.g., a stylus or a mouse), or a hardware button, etc. In addition, when the input accepting unit 150 accepts inputs made by sound, it may also be a microphone.

[0126] [3. Action Example]

[0127] Next, an operation example of the estimation system 200 according to the embodiment will be described.

[0128] [Action Example 1]

[0129] First, refer to Figure 2 Next, operation example 1 of the estimation system 200 according to the embodiment will be specifically described. Figure 2 This is a flowchart showing an operation example 1 of the estimation system 200 according to the embodiment.

[0130] Although in Figure 2 Although not shown, in the estimation system 200 , for example, the sound collection device 10 collects work sounds accompanying the work of the worker and outputs the collected work sound data to the estimation device 100 .

[0131] The acquisition unit 121 of the estimation device 100 acquires data of the working sound collected by the sound collection device 10 ( S01 ), and outputs the acquired data of the working sound to the estimation unit 122 .

[0132] Next, the estimation unit 122 of the estimation device 100 inputs the data of the operation sound into the first model 132 that has been learned, thereby estimating whether the worker is performing an operation of handling a transparent object ( S02 ).

[0133] Hereinafter, step S02 will be described in detail. Figure 3 It indicates schematically Figure 2 For example, the inference unit 122 divides the sound data (i.e., the data of the working sound) obtained from the acquisition unit 121 into data at predetermined intervals (e.g., 2 seconds), and inputs the divided data into the sound recognition model (e.g., the first model 132). Figure 3As shown, the data of the working sound may be subjected to pre-processing such as normalization before being input into the sound recognition model. The sound recognition model extracts the feature quantity of the working sound of the transparent object processing operation from the input working sound data. Here, the feature quantity extracted by the sound recognition model is referred to as the feature quantity of the evaluation object, that is, the evaluation sound feature quantity.

[0134] Next, the inference unit 122 calculates the similarity indicating the degree of similarity between the evaluation sound feature quantity output from the sound recognition model and the feature quantity of the operation sound of the operation of processing transparent objects (here, referred to as the target sound) pre-registered in the storage unit 130, that is, the registered feature quantity, and outputs the calculated similarity.

[0135] Figure 4 FIG. 1 is a graph showing the similarity between the feature value of the collected working sound and the feature value of the working sound of the work of handling a transparent object. Figure 4 It also records the result obtained by the user visually confirming the image taken by the camera device 20 and distinguishing the interval in which the operator is performing the operation of processing transparent objects (here, referred to as the operation interval) from the interval in which the operator is not performing the operation of processing transparent objects (here, referred to as the non-operation interval). In addition, the dotted line in the figure represents the threshold of the similarity. For example, when the similarity of the feature value of the operation sound extracted by the sound recognition model to the feature value of the operation sound of the operation processing operation registered in advance is greater than the threshold value (here 30), it is inferred that the operator is performing the operation of processing transparent objects. Figure 4 As shown, the difference between the working section and the non-working section is represented by the similarity score. For example, if the sound generated in the operation of handling a transparent object (e.g., a plastic bag, a cushioning material, etc.) is collected, the similarity score increases. On the other hand, in the section where a transparent object is placed on the workbench but the worker does not touch the transparent object, since the sound emitted from the transparent object is not collected, the similarity score is not calculated.

[0136] [Verification Example 1 of Action Example 1]

[0137] Next, a description will be given of Verification Example 1 which verifies the estimation accuracy of the work of Operation Example 1. In Verification Example 1, one hour of work sounds are analyzed in a time series. Figure 5 1 is a diagram showing the result of analyzing the work sound of one hour in time series in Verification Example 1. In Verification Example 1, the transparent object is a transparent plastic bag (hereinafter referred to as a transparent bag), and the data of the work sound accompanying the work of the operator is input into the first model 132 (for example, Figure 3The similarity between the feature quantity of the operation sound obtained by the sound recognition model of the sound recognition model and the feature quantity of the operation sound associated with the operation of handling the transparent bag registered in advance. In addition, the data of the operation sound collected in the verification example 1 is the data of the sound in the audible frequency band, but it can also include the data of the sound in the non-audible frequency band.

[0138] and Figure 4 Similarly, the user visually checks the image and manually assigns a correct answer label indicating that a transparent bag processing operation (hereinafter also referred to as bag operation) is being performed. Figure 5 In the example, the state where the worker does not touch the transparent bag but the transparent bag exists on the workbench and the state where the worker is bagging the goods are set as "bag work" and given the correct answer label. On the other hand, the state where the worker is registering documents and the work of removing packaging are set as work that does not handle transparent bags (i.e., non-bag work).

[0139] also, Figure 5 The similarity of the feature amount of the image shown shows the similarity between the feature amount representing the transparent bag appearing in the image extracted using the image recognition model and the feature amount representing the transparent bag appearing in the image registered in advance.

[0140] like Figure 5 As shown, when a sound other than the sound emitted from the transparent bag is generated, the similarity score increases. In Verification Example 1, the recognition accuracy of the task obtained by the sound recognition model is 28% for the correct answer rate and 5% for the incorrect answer rate.

[0141] In addition, in the verification example 1 of the action example 1, an example of the first model 132 that calculates the similarity and estimates the transparent object and an example of the flow of its action are described, but the present invention is not limited to this example. For example, the first model 132 may also be an operation model that directly estimates (in other words, outputs) whether the operation sound data is a transparent object. The following describes another example of the first model 132 and an example of the flow of its action.

[0142] [Verification Example 2 of Action Example 1]

[0143] Next, verification example 2 of operation example 1 will be described. In verification example 2 of operation example 1, an example will be described in which the first model 132 is a model that inputs work sound data and directly outputs an inference result of whether the work is to handle a transparent object. Figure 6 This is a diagram for explaining the estimation method of the bag operation executed in Verification Example 2. Figure 6 The neural network shown is an example of the first model 132 .

[0144] First, the learning of the neural network used for estimating the bag operation will be described.

[0145] The model generation unit 140 uses, as learning data, an image of the frequency spectrum of the work sound or image data corresponding to the work sound (i.e., taken at the same time as when the work sound was collected) showing the worker. In addition, the model generation unit 140 uses, as teacher data, data in which labels are assigned to the learning data in two categories, i.e., whether the worker is doing bag work (in other words, whether there is bag work), or in the case of bag work, in three categories, i.e., the type of the bag (e.g., large bag, small bag, etc.). The model generation unit 140 determines the parameters of the neural network through learning.

[0146] Next, the inference unit 122 performs inference using the neural network using the parameters determined during learning. For example, the inference unit 122 inputs the data for the operation classification (operation sound data or image data) into the neural network, and outputs the inference result of two classifications of the presence or absence of bag operation or three classifications of the type of bag when bag operation exists.

[0147] Figure 7 Yes means Figure 6 An example of the architecture of a neural network is shown in FIG. Figure 7 In the example of , the neural network has a convolutional layer because the input data is an image, but it may not have a convolutional layer if the input data is time-series numerical data. Figure 7 The example is just an example and is not limited to this.

[0148] [Speculations about the two categories]

[0149] First, the estimation of two categories of the presence or absence of bag operation will be described. Figure 8 This is a diagram showing the calculation method of the correct answer rate when making inferences on two categories. The neural network is trained using data with labels of bag-in-hand tasks and bag-out-hand tasks as teacher data. The correct answer rate (%) is calculated by Figure 8 The calculation is performed using the formula shown in the figure. Fig. 9 The inference results and correct answer rates are shown in .

[0150] Fig. 9 This is a diagram showing the inference results of two categories and the correct answer rates in Verification Example 2 of Action Example 1. Fig. 9 (a) shows the inference results and correct answer rates of two categories when the data of the working sound input to the neural network is the data of the sound in the audible frequency band, Fig. 9 (b) shows the inference results and correct answer rates of two categories when the operation sound data is broadband sound data including non-audible frequency band sound. Bag operation 1 is an operation of processing polyethylene bags with a length and width of about 10 cm, and bag operation 2 is an operation of processing polyethylene bags with a length and width of about 30 cm. Fig.19 (a) and Fig. 9 As shown in (b), if the work sound data of wide-band sound including sound in the non-audible band is used as input data, the correct answer rate is higher than that of the work sound data of the audible band. Therefore, when it is confirmed that the work sound data is sound data with a wider frequency band than the sound data of the audible band, the work of the operator can be estimated with higher accuracy.

[0151] [Speculations about the three categories]

[0152] Next, the estimation of three classifications for classifying the types of bags when there is bag operation will be described. Fig.10 This is a diagram showing how to calculate the correct answer rate when making inferences about three categories. The neural network was trained using data with labels indicating the type of bag in the case of bag operations and labels for bagless operations as teacher data. The correct answer rate (%) is given by Fig.10 The calculation is performed using the formula shown in the figure. Fig.11 The inference results and correct answer rates are shown in .

[0153] Fig.11 This is a diagram showing the inference results of three categories and the correct answer rates of Verification Example 2 of Action Example 1. Fig.11 (a) shows the inference results and correct answer rates of three categories when the data of the working sound input to the neural network is the data of the sound in the audible frequency band. Fig.11 (b) shows the estimation result and correct answer rate when the data of the operation sound is data of a wide-band sound including a sound in an inaudible frequency band. Fig.11 (a) and Fig.11 As shown in (b), if the work sound data of wide-band sound including sound in the non-audible band is used as input data, the correct answer rate is higher than that of the work sound data of the audible band. Therefore, when it is confirmed that the work sound data is sound data with a wider frequency band than the sound data of the audible band, the work of the operator can be estimated with higher accuracy.

[0154] [Speculation about two classifications based on the combination of input data]

[0155] Next, the estimation of two categories of the presence or absence of bag operation based on the combination of input data will be described. Fig.12 This is a diagram showing a method of estimating two categories based on a combination of input data and a method of calculating the correct answer rate. Fig.12 (a) represents the classification method of the inference results, Fig.12 (b) shows the correspondence between the inference result and the label. Fig.12In (a), category A indicates that the operation can be correctly inferred according to the label of the bag-existing operation when the input data is at least one of (i) image data and (ii) image data + broadband sound data. In addition, category D indicates that the operation can be correctly inferred according to the label of the bag-free operation when the input data is at least one of the above (i) and (ii). The correct answer rate (%) is given by Fig.12 Calculate using the formula shown in (b). Fig.13 The inference results and correct answer rates are shown in .

[0156] Fig.13 This is a diagram showing the estimation results of two classifications based on the combination of input data and the correct answer rate in Verification Example 2 of Operation Example 1. Fig.13 (a) shows the inference results and correct answer rates of two classifications when the input data to the neural network is image data. Fig.13 (b) shows the estimation results and correct answer rates when the input data is image data and broadband working sound data. Fig.13 (a) and Fig.13 As shown in (b), when broadband work sound data is used as input data, the correct answer rate is higher than when only image data is used. Therefore, it is confirmed that the worker's work can be estimated more accurately when image data and broadband work sound data are used as input data to the neural network, compared with only image data.

[0157] [Verification Example 3 of Action Example 1]

[0158] Next, Verification Example 3 of Action Example 1 is described in detail. In Verification Example 1, work sounds in the audible frequency band are used in the estimation of the work, while in Verification Example 3, it is different from Verification Example 1 in that data of work sounds including sounds in the non-audible frequency band are used. Furthermore, in Verification Example 3, the estimation accuracy when the estimation method described in Action Example 1 is implemented using data of work sounds including sounds in the non-audible frequency band (referred to as this method) is compared with the estimation accuracy when the estimation method using image AI (in other words, video AI) is implemented. In addition, the image AI uses ordinary image AI. The results are shown in Fig.14 middle.

[0159] Fig.14 This is a graph showing the comparison results of the estimation accuracy of the estimation method using image AI and the estimation accuracy of this method. Fig.14In the label column, "1" indicates a label indicating that a transparent bag processing operation (i.e., bag operation) is being performed (i.e., the correct answer label), and "0" indicates that there is no correct answer label (i.e., non-bag operation). In addition, "1" in the image AI and the present method columns indicates that a bag operation is being performed, and "0" indicates that a bag operation is not being performed. Here, it was confirmed whether the "0" and "1" recorded in the label column are consistent with the inference results based on the image AI and the present method. As a result, the inference accuracy of the image AI is 0%, and the inference accuracy of the present method is 72%.

[0160] The results of Verification Example 3 confirmed that by using data of work sounds including sounds in the non-audible frequency band to estimate the work, the estimation accuracy of the work of processing transparent objects is improved compared with sounds in the audible frequency band. In addition, it was confirmed that by using image AI (i.e., image recognition model) and sound recognition model together, the estimation accuracy is improved compared with the case where only image AI is used to estimate the work.

[0161] [Action Example 2]

[0162] Next, refer to Fig.15 , Fig.16 and Fig.17 Next, operation example 2 of the estimation system 200 according to the embodiment will be specifically described. Fig.15 This is a diagram for explaining the difference between the estimation result using the data of the working sound and the estimation result using the data of the image. Fig.16 This is a diagram for explaining an outline of the flow of operation example 2 of the estimation system 200 according to the embodiment. Fig.17 1 is a flowchart showing an operation example 2 of the estimation system 200 according to the embodiment. In the operation example 2, the differences from the operation example 1 are mainly described, and the description of the common steps is omitted or simplified.

[0163] First, the origin of action example 2 is explained. Fig.15 As shown, the user visually confirms the operator's work in the image and determines the interval where the bag operation exists (bag operation interval), and confirms the difference between the visual judgment result, the estimated result of the bag operation based on the operation sound (the similarity of the operation sound to the bag operation sound), and the estimated result of the bag operation based on the image. The number of bag operations can be counted even when the operation is estimated using only image data. However, for example, the score of the similarity of the sound reacts (increases) before the estimation based on the image. In addition, for example, because the transparent bag is blocked by documents and the like and does not appear in the image, the bag operation cannot be estimated even if the image data is used, but because there is a sound (transparent bag sound) emitted when the transparent bag is handled, the score of the similarity of the sound reacts (increases).

[0164] Thus, in the estimation of bag operation based on the image, when the transparent bag is not shown in the image, it is sometimes impossible to estimate that the worker is performing bag operation. Therefore, by combining the estimation of bag operation based on the image with the estimation of bag operation based on the operation sound to estimate bag operation, it is possible to estimate bag operation with higher accuracy.

[0165] Next, the outline of the process of action example 2 is described. Fig.16 As shown in the example 2 of operation, if the image data corresponding to the work sound data is acquired, the acquisition unit 121 of the estimation device 100 inputs the image data to the estimation system 200 that takes the image data as input. Figure 3 As shown, the input image data is pre-processed such as resizing or standardizing, and is input into a neural network (e.g., an image recognition model). Based on the feature quantity of the output image, the similarity with the feature quantity representing the transparent bag appearing in the image is calculated. In addition, in operation example 2, if the data of the working sound is obtained, the acquisition unit 121 of the estimation device 100 inputs the data of the working sound to the estimation system 200 that takes the data of the working sound as input. For example, the system Figure 3 As shown, the input operation sound data is normalized and other pre-processed, and then input into a neural network (e.g., a voice recognition model) and based on the feature quantity of the output operation sound, the similarity with the feature quantity of the bag operation sound is calculated. Then, the inference results obtained by the inference system 200 are combined to output the inference results.

[0166] Next, refer to Fig.17 While explaining action example 2. Fig.17 Although not shown in the figure, the sound collection device 10 collects the work sounds accompanying the work of the worker and outputs the collected work sound data to the estimation device 100. In addition, the camera device 20 captures an image corresponding to the work sound collected by the sound collection device 10 (that is, captured at the same time) showing the worker performing the work, and outputs the captured image data to the estimation device 100. In addition, in the image, when the worker is performing the work of handling a transparent object, the transparent object (here, a transparent bag) appears in the image together with the worker.

[0167] Next, if the data of the operation sound accompanying the operation of the operator is obtained (S01), the acquisition unit 121 of the estimation device 100 outputs the obtained operation sound data to the estimation unit 122. Next, the estimation unit 122 estimates whether the operator is performing the operation of handling a transparent object by inputting the operation sound data into the first model 132 (S02). Specifically, for example, when the similarity between the feature amount extracted by the first model 132 and the feature amount of the operation sound of the operation of handling a transparent object stored in advance in the storage unit 130 is greater than a predetermined value (i.e., a threshold), the estimation unit 122 estimates that the operator is performing the operation of handling a transparent object.

[0168] Furthermore, if data of an image showing a worker performing a work corresponding to the work sound data is obtained (S03), the acquisition unit 121 of the estimation device 100 outputs the acquired image data to the estimation unit 122. Next, the estimation unit 122 estimates whether the worker is performing a work of handling a transparent object by inputting the image data into the second model 133 (S04). Specifically, for example, when the similarity between the feature quantity of the image showing the worker performing a transparent work extracted by the second model 133 and the feature quantity of the image showing the worker performing a transparent work stored in the storage unit 130 in advance is greater than a predetermined value (i.e., a threshold value), the estimation unit 122 estimates that the worker is performing a work of handling a transparent object.

[0169] Next, the inference unit 122 infers whether the worker is performing a work of handling a transparent object based on the inference result inferred from the data of the work sound using the first model 132 and the inference result inferred from the data of the image using the second model 133 (S05). Specifically, for example, the inference unit 122 infers that the worker is performing a work of handling a transparent object when the similarity between the feature quantity of the work sound extracted by the first model 132 and the feature quantity of the work sound of the work of handling a transparent object stored in advance in the storage unit 130 is greater than a predetermined value (threshold value), and the similarity between the feature quantity of the image extracted by the second model 133 and the feature quantity of the image showing the worker handling a transparent object stored in advance in the storage unit 130 is greater than a predetermined value (threshold value), the worker is performing a work of handling a transparent object.

[0170] [Modification 1 of Operation Example 2]

[0171] In Action Example 2, an example of estimating whether the operator is performing an operation of handling a transparent object is described based on a feature amount obtained by inputting operation sound data to the first model 132 and a feature amount obtained by inputting image data to the second model 133. In Modification Example 1 of Action Example 2, by using the example of the first model 132 directly estimating whether the operation is handling a transparent object described in Verification Example 2 of Action Example 1, it is estimated whether the operator is performing an operation of handling a transparent object based on a feature amount of operation sound obtained by inputting operation sound data and image data to the first model 132 and a feature amount of the image.

[0172] Fig.18 1 is a flowchart showing a modification example 1 of the operation example 2 of the estimation system 200 according to the embodiment. Fig.18 As shown, the acquisition unit 121 of the estimation device 100 acquires data of the working sound collected by the sound collection device 10 (S01), and outputs the acquired data to the estimation unit 122. In addition, the acquisition unit 121 of the estimation device 100 acquires data of an image showing a worker performing a work, which corresponds to the data of the working sound, and is captured by the camera device 20 (S03), and outputs the acquired data to the estimation unit 122.

[0173] Next, the estimation unit 122 estimates whether the worker is performing an operation of handling a transparent object based on the feature amount of the operation sound and the feature amount of the image obtained by inputting the data of the operation sound and the data of the image into the first model 132 ( S06 ).

[0174] [Configuration Example 1 of the Estimation Unit 122 for Executing the Flow of Modification Example 1 of Operation Example 2]

[0175] Next, a configuration example 1 of the estimation unit 122 that executes the flow of the modification example 1 of the operation example 2 is described. Fig.19 This is a diagram schematically showing a configuration example 1 of the estimation unit 122 that executes the flow of the modification example 1 of the operation example 2. Fig. 20 This is a diagram for explaining the estimation method of the bag operation executed by the configuration example 1.

[0176] like Fig.19 As shown, the estimation unit 122 includes an embedding vector generation unit, a task classification unit, and a bag task determination unit. The embedding vector generation unit includes an image sub-network that extracts image features by taking image data as input, a sound sub-network that extracts sound features by taking sound (here, task sounds) data as input, and a fusion layer.

[0177] like Fig.19As shown, the neural network may include, for example, an image subnetwork and a sound subnetwork. Such a neural network may be the first model 132. In addition, the sound subnetwork may be the first model 132, and the image subnetwork may be the second model 133.

[0178] like Fig. 20 As shown, the model generation unit 140 uses image data and work sound data as learning data, and uses data to which a label indicating whether the learning data is similar is given as teacher data. The model generation unit 140 determines the parameters of the neural network through learning. The work sound data is data of a wide-band sound including sounds in an audible band or sounds in an inaudible band. The work sound data may be, for example, a spectrum of 257×199 pixels. The image data may be, for example, data of 224×224 pixels. In addition, the model generation unit 140 may perform transfer learning on the fusion layer.

[0179] Next, the estimation unit 122 creates an embedding vector by using a fusion layer using the parameters determined during learning. Next, the estimation unit 122 inputs the embedding vector to the job classification unit, and determines the bag job based on the probability value output from the Softmax layer.

[0180] [Configuration Example 2 of the Estimation Unit 122 for Executing the Flow of Modification Example 1 of Operation Example 2]

[0181] Next, a configuration example 2 of the estimation unit 122 that executes the flow of the modification example 1 of the operation example 2 is described. Fig.21 This is a diagram of a configuration example 2 of the inference unit 122 schematically showing the flow of the modification example 1 of the execution example 2. In the configuration example 1, the job classification unit has a classification network and a Softmax layer, while in the configuration example 2, the job classification unit has a contrastive learning network. Contrastive learning is a type of self-supervised learning that can directly learn a large amount of data by comparing data with each other without assigning labels. In contrastive learning, feature quantities are learned in such a way that similar data are close and different data are far apart.

[0182] [Configuration Example 3 of the Estimation Unit 122 for Executing the Flow of Modification Example 1 of Operation Example 2]

[0183] Next, a configuration example 3 of the estimation unit 122 that executes the flow of the modification example 1 of the operation example 2 will be described. Fig. 22 This is a diagram schematically showing a configuration example 3 of the estimation unit 122 that executes the flow of the modification example 1 of the operation example 2. In the configuration examples 1 and 2, the fusion layer is arranged before the classification network, while in the configuration example 3, the fusion layer is arranged after the classification network.

[0184] [Examples of the architecture of the image subnetwork and the sound subnetwork]

[0185] Next, examples of the architecture of the image sub-network and the audio sub-network are described. Fig.23 This is a diagram showing an example of the architecture of the image sub-network. Fig.24 is a diagram showing an example of the architecture of the sound sub-network. Fig.23 and Fig.24 As shown, since the sizes of the image data and the work sound data used as input data are often different, the sizes of each layer of the image subnetwork and the sound subnetwork may also be different, as long as the sizes of the final layers of these subnetworks are the same.

[0186] [Example of the architecture of the fusion layer]

[0187] Fig.25 is a diagram showing an example of the architecture of the fusion layer. Fig.25 As shown in the figure, the data output from the image sub-network and the data output from the sound sub-network are input into the combination layer, and different outputs are obtained during learning and inference.

[0188] [Example of the architecture of a classification network]

[0189] Fig.26 is a diagram showing an example of the architecture of a classification network. Fig.26 As shown, the size of the first layer of the classification network is the same size as the data output from the final layer of the image subnetwork when it is configured after the image subnetwork, and the same size as the data output from the final layer of the sound subnetwork when it is configured after the sound subnetwork. In addition, the size of the first layer of the classification network is the same size as the data output from the final layer of the fusion layer when the classification network is configured after the fusion layer.

[0190] [Example of contrastive learning network architecture]

[0191] Fig. 27 is a diagram showing an example of the architecture of a contrastive learning network. Fig. 27 As shown, the size of the first layer of the contrastive learning network is, for example, the same as the size of the embedding vector output from the embedding vector production unit. The contrastive learning network is used for transfer learning. However, as a loss function, for example, the following formula 1 is used.

[0192] [Formula 1]

[0193]

[0194] Here, sim(x, y) is a function for calculating similarity, for example, cosine similarity can be used. zi, zj are corresponding embedding vectors, for example, embedding vectors of image data and broadband working sound data can be used respectively. τ is an adjustment parameter.

[0195] The loss function of the above formula 1 becomes larger when the similarity between the two embedding vectors is large, and becomes smaller when the similarity is small.

[0196] [Configuration Example of the Estimation Unit 122 for Executing the Process of Modification Example 2 of Operation Example 2]

[0197] Fig.28 1 is a diagram schematically showing a configuration example of the estimation unit 122 for executing the process of the modified example 2 of the operation example 2. Fig.18 The process of the modification example 1 of the described operation example 2 is described. In the modification example 1 of the operation example 2, the image data is obtained and used as the input data, and in the modification example 2 of the operation example 2, Fig.18 In step S03 of the present invention, instead of the image data captured by the camera device 20, distance data acquired by a distance measuring sensor or the like may be used as input data. Fig.28 As shown, the estimation unit 122 has a distance sub-network instead of an image sub-network. Fig.28 In the example, for example, the configuration position of the fusion layer may be changed as in Structure Example 2 or Structure Example 3 of Variation Example 1 of Action Example 2, and the job classification unit may have a contrastive learning network.

[0198] [Action Example 3]

[0199] Next, refer to Fig.29 , Fig. 30A and Fig. 30B Next, operation example 3 of the estimation system 200 according to the embodiment will be described. Fig.29 This diagram shows an example of a work sound in which the estimation unit 122 mistakenly estimates that the worker is performing a work of handling a transparent object. Fig. 30A This is a flowchart of operation example 3 of the estimation system 200 according to the embodiment. Fig. 30B This is a flowchart showing an example of an operation of preregistering feature amounts of working sounds that may be erroneously estimated.

[0200] In Operation Example 3, the operation sound accompanying the operation of handling transparent bags is called transparent bag sound, and the operation sound accompanying the operation of handling non-transparent bags (in other words, not handling transparent bags) is called non-transparent bag sound. In addition, the operation of handling transparent bags is called bag operation.

[0201] First, refer to Fig.29 The following describes the operation sound that may be misjudged (hereinafter, also referred to as misjudged target sound). Fig.29In the example of , the threshold of the similarity is, for example, 25. When the similarity of the operation sound to the transparent bag sound is greater than the threshold, the inference unit 122 infers that the operator is performing an operation of handling a transparent bag (bag operation). At this time, there are cases where the operator's operation is correctly inferred based on the operation sound accompanying the operation of handling a transparent bag, such as the sound of opening a plastic bag or the sound of taking a bag out of a shelf, but there are also cases where the operator is mistakenly inferred to be performing a bag operation even though the bag operation is not being performed. For example, based on operation sounds other than the transparent bag sound (i.e., non-transparent bag sounds), such as the sound of tying boxes with rubber bands, the sound of storing boxes or bags in the lower section of a cart, or the sound of scanning a barcode while moving, it is mistakenly inferred that the operator is performing a bag operation.

[0202] In order to reduce such misjudgment, the estimation unit 122 calculates the similarity between the feature quantity of the non-transparent bag sound and the feature quantity of the pre-registered transparent bag sound, and when the similarity is higher than a threshold, the non-transparent bag sound is determined as a misjudged target sound and stored in the storage unit 130. In operation example 3, the estimation unit 122 reads the feature quantity of the pre-registered operation sound that may be misjudged (hereinafter, also referred to as the misjudged target sound) and the feature quantity of the transparent bag sound from the storage unit 130, compares the feature quantity of the operation sound with the similarity of these feature quantities, and estimates whether the operator is performing bag operation.

[0203] Next, refer to Fig. 30A Operation Example 3 will be described. Although not shown in the figure, the acquisition unit 121 of the estimation device 100 acquires data of the working sound collected by the sound collection device 10 , and outputs the acquired data to the estimation unit 122 .

[0204] The estimation unit 122 inputs the acquired working sound data into the voice recognition model ( S11 ), detects the voice based on the input working sound data, and extracts the input feature amount ( S12 ).

[0205] Next, the estimation unit 122 extracts the feature value (sound feature value) of the working sound (hereinafter referred to as the input sound) using the sound recognition model (S13). Next, the estimation unit 122 reads the feature value of the transparent bag sound and the feature value of the erroneously estimated sound from the storage unit 130 (S14).

[0206] Next, the estimation unit 122 calculates the similarity between the transparent bag sound and the input sound and the similarity between the erroneously estimated target sound and the input sound in similarity calculation ( S15 ).

[0207] Next, the estimation unit 122 determines whether the similarity between the transparent bag sound and the input sound is higher than the similarity between the erroneously estimated object sound and the input sound (S16). If it is determined that it is higher (“Yes” in S16), it determines whether the similarity between the transparent bag sound and the input sound is higher than a threshold value (S17). If it is determined that the similarity between the transparent bag sound and the input sound is higher than the threshold value (“Yes” in S17), the estimation unit 122 determines that the input sound is a transparent bag sound (S18). Thus, the estimation unit 122 estimates that the worker is performing the operation of handling a transparent bag based on the feature amount of the input sound (operation sound).

[0208] On the other hand, when the estimation unit 122 determines in step S16 that the similarity between the transparent bag sound and the input sound is not higher than the similarity between the erroneously estimated object sound and the input sound (No in S16), it determines that the input sound is not a transparent bag sound (S19). In addition, when the estimation unit 122 determines in step S17 that the similarity between the transparent bag sound and the input sound is not higher than the threshold value (No in S17), it determines that it is not a transparent bag sound (S19). Thus, the estimation unit 122 estimates that the worker is performing a work without handling a transparent bag based on the feature amount of the input sound (work sound).

[0209] Next, refer to Fig. 30B While explaining an operation example in which the feature amount of the erroneously estimated object sound used in operation example 3 is stored in advance in the storage unit 130. Although not shown in the figure, the acquisition unit 121 of the estimation device 100 acquires the data of the operation sound acquired by the sound collection device 10, and outputs the acquired data to the estimation unit 122. At this time, the data of the operation sound acquired by the acquisition unit 121 is the operation sound accompanying the operation of not handling the transparent bag.

[0210] Next, the estimation unit 122 inputs the acquired working sound data into the voice recognition model ( S21 ), detects the voice based on the input working sound data, and extracts the input feature amount ( S22 ).

[0211] Next, the estimation unit 122 extracts the feature amount (sound feature amount) of the working sound (hereinafter referred to as the input sound) using the sound recognition model (S23). Next, the estimation unit 122 reads the feature amount of the transparent bag sound from the storage unit 130 (S24).

[0212] Next, the estimation unit 122 calculates the similarity between the transparent bag sound and the input sound in similarity calculation ( S25 ).

[0213] Next, the estimation unit 122 determines whether the similarity between the transparent bag sound and the input sound is higher than a threshold value (S26). If it is determined that it is higher than the threshold value ("Yes" in S26), the input sound is determined to be a falsely estimated object sound (S27). In addition, the estimation unit 122 stores the feature amount of the collected sound (work sound) in the storage unit 130 as the feature amount of the falsely estimated object sound (S29). On the other hand, if it is determined that the similarity between the transparent bag sound and the input sound is not higher than the threshold value ("No" in S26), the estimation unit 122 determines that the input sound is not a falsely estimated object sound (S28).

[0214] [5. Effects, etc.]

[0215] As described above, the estimation method according to the present embodiment is an estimation method for estimating the work of an operator performed by a computer (e.g., estimation device 100), wherein the computer obtains data (e.g., the work sound data) associated with the work. Figure 2 S01 of FIG. 1 ), by inputting the data of the operation sound into the first model 132 that has been learned, it is inferred whether the operator is performing the operation of handling a transparent object ( Figure 2 of S02).

[0216] Thus, the device executing the inference method (for example, the inference device 100) uses the first model 132 that takes the work sound data as input and outputs whether the work is to process a transparent object, so it can accurately infer the work of processing a transparent object.

[0217] For example, in the estimation method according to the present embodiment, a computer (for example, the estimation device 100) obtains data ( Fig.17 S03 of FIG. 10 ), the image data is input to the learned second model 133, thereby inferring whether the operator is processing a transparent object ( Fig.17 S04), based on the inference result using the first model 132 and the inference result using the second model 133, it is inferred whether the operator is performing an operation of handling a transparent object ( Fig.17 In addition, the inference result using the first model 132 is the inference result inferred by the first model 132 based on the data of the work sound, and the inference result using the second model 133 is the inference result inferred by the second model 133 based on the data of the image.

[0218] Thus, the device executing the estimation method (e.g., the estimation device 100) estimates whether the worker is performing the operation of handling a transparent object based on the estimation result estimated by the first model 132 based on the data of the operation sound and the estimation result estimated by the second model 133 based on the data of the image. Therefore, the device executing the estimation method can estimate the operation of handling a transparent object with higher accuracy than when the estimation is performed using only the data of the operation sound.

[0219] For example, in the estimation method according to the present embodiment, a computer (for example, the estimation device 100) obtains data ( Fig.18 In S03), by inputting the data of the working sound and the image data into the first model 132, it is estimated whether the operator is performing the operation of handling a transparent object (S06).

[0220] Thus, the device that executes the inference method (for example, the inference device 100) uses the first model 132 that takes the data of the working sound and the data of the image corresponding to the working sound as input and outputs whether the operation is to process a transparent object. Therefore, compared with the case where only the data of the working sound is used for inference, the operation of processing a transparent object can be inferred with higher accuracy.

[0221] For example, in the estimation method according to the present embodiment, the computer (e.g., the estimation device 100) generates an estimation result based on the feature amount of the work sound output from the first model 132 and the feature amount of the work sound pre-stored in the storage unit 130 (e.g., Figure 1 Based on the similarity of the feature quantities of the work sounds of the work of handling transparent objects in the feature quantity database 131), it is inferred whether the worker is performing the work of handling transparent objects.

[0222] Thus, the device that executes the inference method (for example, the inference device 100) infers whether the worker is performing the operation of processing a transparent object based on the similarity between the characteristic quantity of the working sound output from the first model 132 and the characteristic quantity of the working sound of the operation of processing a transparent object, so that the operation of processing a transparent object can be inferred with high accuracy.

[0223] For example, in the inference method related to the present embodiment, the computer (e.g., the inference device 100) determines the similarity (in other words, the first similarity) of the feature quantity of the work sound output from the first model 132 to the feature quantity of the work sound of the work of processing a transparent object pre-stored in the storage unit 130 (e.g., the feature quantity database 131), and the work sound that may be mistakenly inferred as the worker performing the work of processing a transparent object (e.g., the feature quantity database 131) pre-stored in the storage unit 130 (e.g., the feature quantity database 131). Fig. 30AThe similarity of the feature quantity of the falsely estimated object sound (in other words, the second similarity) is used to infer whether the operator is performing an operation of handling a transparent object ( Fig. 30A S16~S19).

[0224] Thus, the device executing the estimation method (e.g., the estimation device 100) can reduce the occurrence of mis-estimation by comparing the similarity (first similarity) between the feature quantity of the operation sound output from the first model 132 and the feature quantity of the operation sound of the operation of handling a transparent object, and the similarity (second similarity) between the feature quantity of the operation sound output from the first model 132 and the feature quantity of the operation sound that may be mis-estimated. Therefore, the device executing the estimation method can accurately estimate the operation of handling a transparent object even if only the data of the operation sound is used.

[0225] For example, in the estimation method according to the present embodiment, the computer (e.g., the estimation device 100) determines the similarity (the first similarity) of the feature quantity of the working sound output from the first model 132 to the feature quantity of the working sound of the operation of handling a transparent object higher than the similarity (the first similarity) of the feature quantity of the working sound output from the first model 132 to the feature quantity of the working sound of the operation of handling a transparent object. Fig. 30A In the case of the above-mentioned similarity (the above-mentioned second similarity) of the feature quantity of the erroneously estimated target sound) ( Fig. 30A ), it is inferred that the operator is performing an operation of handling a transparent object.

[0226] Thus, the device executing the estimation method (for example, the estimation device 100 ) can reduce the occurrence of erroneous estimation, and thus can estimate the operation of handling a transparent object with high accuracy even using only the data of the operation sound.

[0227] For example, in the inference method according to the present embodiment, when the computer (e.g., the inference device 100) inputs data of the working sound of the working process of processing a non-transparent object different from the transparent object into the first model 132 and the similarity (in other words, the third similarity) of the feature quantity of the working sound of the working process of processing the non-transparent object and the feature quantity of the working sound of the working process of processing the transparent object is higher than the threshold value ( Fig. 30B The operation sound of the operation for processing the non-transparent object is determined to be an operation sound that may be misjudged as the operation sound for processing the transparent object (i.e., the misjudged object sound) ( Fig. 30B The feature quantity of the operation sound of the operation of processing the non-transparent object is stored in the storage unit 130 (for example, the feature quantity database 131) as the feature quantity of the operation sound that may be misjudged ( Fig. 30B S29).

[0228] Thus, the device (e.g., the inference device 100) that performs the inference method can accurately determine whether the work sound of the work of processing the non-transparent object is a work sound that may be misjudged as the work of processing the transparent object based on the similarity (third similarity) between the feature amount of the work sound of the work of processing the non-transparent object and the feature amount of the work sound of the work of processing the transparent object. Therefore, the device that performs the inference method can save the feature amount of the work sound that is more likely to be misjudged in the storage unit 130. Therefore, the device that performs the inference method can reduce the occurrence of misjudgment by using the feature amount of the work sound that may be misjudged stored in the storage unit 130, so even if only the work sound data is used, the work of processing the transparent object can be accurately inferred.

[0229] In the estimation method according to the present embodiment, the data of the working sound may include data of sound in an inaudible frequency band.

[0230] Thus, the device (e.g., the inference device 100) that performs the inference method uses the data of the working sound including the sound from the audible frequency band to the sound from the inaudible frequency band to infer whether the operator is performing the operation of handling the transparent object. In this way, since the data of the working sound includes the sound from the inaudible frequency band, the noise from the environment that becomes the cause of the erroneous inference in the data of the working sound decreases, so the device that performs the inference method can improve the inference accuracy of the operation of handling the transparent object. Furthermore, the device that performs the inference method can infer whether the operator is performing the operation of handling the transparent object based on more information than the case of using only the data of the sound in the audible frequency band. Therefore, the device that performs the inference method can infer the operation of handling the transparent object with higher accuracy.

[0231] In addition, the inference device 100 related to the present embodiment is an inference device for inferring the work of an operator, and comprises: an acquisition unit 121 for acquiring data of work sounds accompanying the collected work; and an inference unit 122 for inferring whether the operator is performing the work of handling a transparent object by inputting the data of the work sounds into the first model 132 that has been learned.

[0232] Thus, the estimation device 100 uses the first model 132 that takes the work sound data as input and outputs whether the work is to handle a transparent object, and therefore can estimate the work to handle a transparent object with high accuracy.

[0233] Furthermore, the program according to the present embodiment is a program for causing a computer to execute the above-mentioned estimation method.

[0234] In this way, it is possible to achieve the same effect as the above-mentioned estimation method using a computer.

[0235] (Other embodiments)

[0236] As mentioned above, although embodiment was described, this disclosure is not limited to the said embodiment.

[0237] Fig.31 This is a block diagram showing an example of the functional structure of the inference system of another embodiment. The inference system 200 of the embodiment describes an example in which the inference device 100 is a server device, but the inference device 100 may not be a server device. For example, in the inference system 200a of another embodiment, the inference device 100a may be a fixed-place computer device such as a personal computer. The inference device 100a is different from the inference device 100 in that it has a display unit 160. Only the differences will be described below.

[0238] [Display unit 160]

[0239] The display unit 160 displays, for example, the estimation result. The display unit 160 is, for example, a display device that displays image information including characters, and is, for example, a display including a liquid crystal (LC) panel or an organic EL (Electro Luminescence) panel as a display device.

[0240] In addition, the inference device 100a may include, for example, a sound pickup unit and a camera unit, and more than one may be provided in the workspace 80. The sound pickup unit and the camera unit may be connected to the sound pickup device 10 and the camera device 20 by wired or wireless communication, or may be a single device including the sound pickup device 10 and the camera device 20. Furthermore, the inference device 100a may also be connected to a server device or a user's information terminal for communication, for example. In this case, the inference device 100a may store the inference result in the storage unit 130 for a specified period (for example, 1 day, several days, a week, etc.), output the inference result to the server device or the information terminal, or output the inference result each time the inference is made. The server device may be a cloud server. In addition, the information terminal may be a fixed-placed computer device such as a personal computer, or a portable computer device such as a tablet terminal.

[0241] In addition, for example, in the above-mentioned embodiment, the inference systems 200 and 200a are respectively implemented by a plurality of devices, but they may also be implemented as a single device. In addition, when the system is implemented by a plurality of devices, the plurality of components respectively included in the inference systems 200 and 200a may be distributed to the plurality of devices in any manner. In addition, for example, a server device capable of communicating with the inference system 200 or 200a may include a plurality of components included in the information processing unit 120.

[0242] For example, the communication method between the devices in the above-mentioned embodiment is not particularly limited. In addition, the communication between the devices may be carried out via a relay device not shown in the figure.

[0243] Furthermore, in the above-described embodiment, the processing executed by a specific processing unit may be executed by another processing unit. Furthermore, the order of a plurality of processing may be changed, or a plurality of processing may be executed in parallel.

[0244] In addition, in the above-mentioned embodiments, each component can also be implemented by executing a software program suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0245] In addition, each component may also be implemented by hardware. For example, each component may also be a circuit (or integrated circuit). These circuits may constitute one circuit as a whole, or they may be different circuits. In addition, these circuits may be general circuits or dedicated circuits.

[0246] In addition, all or specific technical solutions of the present disclosure may also be implemented by a system, device, method, integrated circuit, computer program, or a computer-readable CD-ROM or other recording medium. In addition, it may also be implemented by any combination of a system, device, method, integrated circuit, computer program, and recording medium.

[0247] For example, the present disclosure may be implemented as an inference method executed by a computer such as the inference device 100, or as a program that causes a computer to execute such an inference method. In addition, the present disclosure may be implemented as a program for causing a general-purpose computer to act as the inference device 100 of the above-mentioned embodiment. The present disclosure may also be implemented as a computer-readable non-volatile recording medium having these programs recorded thereon.

[0248] In addition, various modifications that can be conceived by those skilled in the art to the embodiments or embodiments achieved by arbitrarily combining the components and functions of the embodiments without departing from the gist of the present disclosure are also included in the present disclosure.

[0249] Industrial Applicability

[0250] According to the present disclosure, since the work of handling a transparent object can be estimated with high accuracy, the work time can be accurately grasped, and the efficiency of the work at the site such as a factory or logistics can be improved.

[0251] Description of symbols

[0252] 10 Sound pickup device

[0253] 20 Camera device

[0254] 50 Information Terminal

[0255] 80 Workspace

[0256] 100, 100a estimation device

[0257] 110 Ministry of Communications

[0258] 120 Information Processing Department

[0259] 121 Acquisition Department

[0260] 122 Speculation Department

[0261] 130 Storage

[0262] 131 Feature Database

[0263] 132 1st Model

[0264] 133 2nd Model

[0265] 140 Model Generation Department

[0266] 150 Input Acceptance Department

[0267] 160 Display unit

[0268] 200, 200a Inference System

Claims

1. A method for estimating a worker's work by a computer, wherein: The computer acquires the collected data of the operation sound accompanying the operation; The computer inputs the data of the operation sound into the first model that has been learned, thereby inferring whether the worker is performing an operation of handling a transparent object.

2. The inference method according to claim 1, wherein: The computer acquires data corresponding to the data of the work sound, and showing an image of the worker performing the work; The computer infers whether the operator is performing an operation of handling the transparent object by inputting the image data into the learned second model; The computer estimates whether the worker is performing an operation of handling the transparent object based on an estimation result using the first model and an estimation result using the second model.

3. The inference method according to claim 1, wherein: The computer acquires data corresponding to the data of the work sound, and showing an image of the worker performing the work; The computer estimates whether the worker is performing the operation of handling the transparent object by inputting the data of the operation sound and the data of the image into the first model.

4. The estimation method according to any one of claims 1 to 3, wherein: The computer estimates whether the worker is performing the operation of handling the transparent object based on the similarity between the feature amount of the operation sound output from the first model and the feature amount of the operation sound of the operation of handling the transparent object stored in advance in the storage unit.

5. The estimation method according to any one of claims 1 to 3, wherein: The computer further infers whether the operator is performing the operation of processing the transparent object based on the similarity of the feature quantity of the work sound output from the first model to the feature quantity of the work sound of the operation of processing the transparent object pre-stored in the storage unit, and the similarity to the feature quantity of the work sound pre-stored in the storage unit that may be mistakenly inferred that the operator is performing the operation of processing the transparent object.

6. The inference method according to claim 5, wherein: The computer infers that the operator is performing the operation of processing the transparent object when the similarity of the feature quantity of the work sound output from the first model to the feature quantity of the work sound of the operation of processing the transparent object is higher than the similarity to the feature quantity of the work sound that may be mistakenly inferred that the operator is performing the operation of processing the transparent object.

7. The inference method according to claim 5, wherein: The computer determines the work sound of the operation of processing the non-transparent object as a work sound that may be misjudged, when the similarity between the feature quantity of the work sound of the operation of processing the non-transparent object and the feature quantity of the work sound of the operation of processing the transparent object obtained by inputting data of the work sound of the operation of processing the non-transparent object different from the transparent object into the first model is higher than a threshold value, and the work sound that may be misjudged is a work sound that may be misjudged as the work sound of the operation of processing the transparent object; The computer stores the feature amount of the work sound of the work of processing the non-transparent object in the storage unit as the feature amount of the work sound that may be erroneously estimated.

8. The estimation method according to any one of claims 1 to 3, wherein: The data of the working sound includes data of sound in a non-audible frequency band.

9. An inference device for inferring an operation of an operator, comprising: an acquisition unit that acquires the collected data of the operation sound accompanying the operation; and The estimation unit estimates whether the worker is performing an operation of handling a transparent object by inputting the data of the operation sound into the first model that has been learned.

10. A program, wherein: A method for causing a computer to execute the estimation method according to any one of claims 1 to 3.

Images