Safety confirmation support system, inference device, machine learning device, safety confirmation support method, inference method, and machine learning method

Abstract

To provide a safety confirmation support system capable of visually confirming that a safety confirmation operation is correctly performed.SOLUTION: This safety confirmation support system 1 is provided with an image photographing device 2 for photographing a person at a prescribed photographing point, and for generating image date D1, a sound generating device 4 for generating warning sound toward the person, and an information processor 6 for determining whether or not a safety confirmation operation is correctly performed by the person U photographed as the image date D1. The information processor 6 includes the image-data acquiring unit 600 that acquires the image D1 from the image capturing device 2 and the information processor 601 that inputs the image D1 to the learning model 12 to generate the safety confirmation determination information D1 including the determination result as to whether the safety confirmation action is correctly performed as the action of the person captured in the image D2. The learning model 12 is a learned model in which the correlative relationship between the image D1 and the safety confirmation determination information D2 is learned by mechanical learning.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a safety confirmation support system, an inference device, a machine learning device, a safety confirmation support method, an inference method, and a machine learning method. [Background technology]

[0002] At work sites in various industries, a safety confirmation action known as pointing and calling has been introduced, in which the operator looks at the area to be checked, points, and speaks in order to prevent oversight of safety confirmations, work errors, etc. Conventionally, as a technology for confirming that safety confirmations are being reliably performed, Patent Document 1 discloses a technique for performing voice recognition processing on the voice spoken by an operator during a safety confirmation action, comparing the content of the operator's speech with the content of the operation requested of the operator, and blocking the operator's operation if there is a mismatch. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-061587 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 only checks the vocalizations made during the safety confirmation action, and therefore it is not possible to check visually whether the person performing the safety confirmation action is, for example, turning their head and looking at the area to be checked, or pointing their finger at the area to be checked.

[0005] In view of the above problems, the present invention aims to provide a safety confirmation support system, an inference device, a machine learning device, a safety confirmation support method, an inference method, and a machine learning method that enable visual confirmation that safety confirmation operations have been performed correctly. [Means for solving the problem]

[0006] In order to achieve the above object, a safety confirmation support system according to one aspect of the present invention is provided. an image data acquisition unit that acquires image data of a person; an information processing unit that inputs the image data acquired by the image data acquisition unit into a learning model to generate safety confirmation determination information including a determination result as to whether or not a safety confirmation action toward a predetermined confirmation point has been correctly performed as the action of the person captured in the image data, The learning model is This is a trained model that has been trained through machine learning to determine the correlation between the image data and the safety confirmation judgment information. [Effects of the Invention]

[0007] According to a safety confirmation support system according to one aspect of the present invention, image data of a person is input into a learning model, and safety confirmation determination information is generated as the action of the person captured in the image data, including a determination result as to whether or not the safety confirmation action toward a predetermined check point was correctly performed. The learning model to which the image data is input is a trained model that has learned the correlation between the image data and the safety confirmation determination information through machine learning, so that it is possible to visually confirm that the safety confirmation action was correctly performed based on the safety confirmation determination information generated by inputting the image data into the learning model.

[0008] Problems, configurations, and effects other than those described above will become apparent from the detailed description of the invention that follows. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an overall configuration diagram showing an example of a safety confirmation support system 1. FIG. [Figure 2] FIG. 2 is a block diagram showing an example of a machine learning device 5. [Figure 3] 1 is a diagram showing an example of learning data 11 and a learning model 12. FIG. [Figure 4] FIG. 2 is a block diagram showing an example of an information processing device 6. [Figure 5] 10 is a data configuration diagram showing an example of an image data storage unit 62. FIG. [Figure 6] FIG. 2 is a functional explanatory diagram showing an example of an information processing device 6. [Figure 7] FIG. 9 is a hardware configuration diagram showing an example of a computer 900. [Figure 8] 10 is a flowchart showing an example of a machine learning method performed by the machine learning device 5. [Figure 9] 1 is a flowchart showing an example of a safety confirmation support method by the safety confirmation support system 1. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. The scope necessary for the explanation to achieve the object of the present invention will be schematically shown, and the scope necessary for explaining the relevant parts of the present invention will be mainly explained, and the parts that are omitted from the explanation will be based on publicly known techniques.

[0011] 1 is an overall configuration diagram showing an example of a safety confirmation support system 1. The safety confirmation support system 1 according to this embodiment functions as a system for supporting a person U, such as a passerby passing through a predetermined passageway or a worker working at a predetermined work location, to correctly perform a safety confirmation operation.

[0012] Examples of passage places include, but are not limited to, intersections and areas near doors within a business or facility. Examples of work places include, but are not limited to, locations where various machines, equipment, vehicles, and other operating objects are operated, and locations where various tasks related to production, transportation, security, public transportation, etc. are performed. In this case, the various tasks include not only actual work but also tasks performed for training and education. Examples of various production tasks include tasks related to material input, assembly, inspection, etc. Examples of various transportation tasks include tasks related to checking the surroundings, inspecting vehicles, and checking cargo, etc. Examples of various security tasks include tasks related to traffic guidance and patrols by police officers or security guards, etc. Examples of various public transportation tasks include tasks related to checking the surroundings, inspecting vehicles, and directing customers, etc. Note that passage places and work places may be outdoors or indoors.

[0013] The safety confirmation action is an action required to prevent oversight of safety checks, work errors, etc., and a representative action is pointing and calling. The safety confirmation action is performed, for example, by appropriately combining turning the head to direct the gaze to the check point, pointing at the check point, and vocalizing the check point. Examples of check points include, but are not limited to, traffic on both sides at intersections or near doors, the closing of doors or shutters from overhead, the status of objects to be operated, the presence of unspecified people, and the presence of unspecified objects.

[0014] The safety confirmation operation requires checking different points depending on the various passage locations and work locations as described above. Note that the safety confirmation operation may be performed for a plurality of check points in a predetermined order. In this embodiment, the safety confirmation operation is performed by pointing and calling out traffic on both sides at an intersection, so the explanation will be centered on the case where the safety confirmation operation is performed in the order of "right, good" and "left, good."

[0015] The safety confirmation support system 1 mainly comprises an image capturing device 2, a projection device 3, a sound generating device 4 (4L and 4R in FIG. 1), a machine learning device 5, and an information processing device 6. Each of the devices 2 to 6 is configured, for example, as a general-purpose or dedicated computer (see FIG. 7 described later), and is connected to a wired or wireless network 7 so that various data can be transmitted and received between them. The number of the devices 2 to 6 and the connection configuration of the network 7 are not limited to the example in FIG. 1 and may be changed as appropriate. In the example in FIG. 1, the image capturing device 2, projection device 3, and sound generating device 4 are illustrated as being installed at one image capturing location, but they may also be installed at multiple image capturing locations.

[0016] The image capturing device 2 is installed at a predetermined capturing point (an intersection in this embodiment), captures an image of a person U at the capturing point, and generates image data D1. The image data D1 is preferably moving image data, but may also be a plurality of still image data captured at a predetermined time interval. The image data D1 is transmitted to the information processing device 6 via the network 7, for example.

[0017] The image capturing device 2 is configured by, for example, a camera (image sensor) having a predetermined resolution (number of pixels), such as a CMOS sensor or a CCD sensor, etc. The image capturing device 2 may also be an infrared camera or a night vision camera.

[0018] The projection device 3 projects the standing position of the person U when performing a safety confirmation action at a predetermined shooting point. The projection light 30 from the projection device 3 is projected onto the ground or floor in the shape of, for example, a circle or a stop line. The projection device 3 is composed of, for example, lighting such as an LED, a projector, etc. The projection device 3 may constantly project the projection light 30, or may project the projection light 30 when various sensors detect the approach of a passerby, a worker, or the like, or the approach of a vehicle, a cart, etc. to the shooting point.

[0019] The sound generating devices 4 (4L, 4R) are installed on the confirmation point side of the shooting point, and generate a warning sound from the confirmation point of the safety confirmation operation toward the person U located at the shooting point. The sound generating device 4 is configured, for example, by a speaker. Note that the sound generating device 4 may also be configured, for example, to perform sound image localization using multiple speakers. In this embodiment, a case will be described in which the sound generating devices 4 are configured by a sound generating device 4L installed at the confirmation point on the left side of the shooting point, and a sound generating device 4R installed at the confirmation point on the right side of the shooting point.

[0020] The machine learning device 5 acquires, for example, image data D1 captured by the image capturing device 2 as learning data 11, and generates a learning model 12 to be used in the information processing device 6 by machine learning. The trained learning model 12 (trained model) is provided to the information processing device 6 via the network 7, a recording medium, or the like.

[0021] The image data D1 acquired as the learning data 11 includes not only data taken when the safety confirmation operation is performed correctly but also data taken when the safety confirmation operation is not performed correctly. Furthermore, the image data D1 acquired as the learning data 11 may include not only data taken at a shooting location in an actual passageway or work location but also data taken in a test environment.

[0022] The information processing device 6 uses the learning model 12 trained by the machine learning device 5 to determine whether or not the safety confirmation action has been correctly performed by the person U photographed as image data D1 by the image capturing device 2. In addition, the information processing device 6 cooperates with the image capturing device 2, the projection device 3, and the sound generating device 4 to prompt the person U to correctly perform the safety confirmation action by projecting light 30 or sounding an alarm.

[0023] (Machine Learning Device 5) 2 is a block diagram showing an example of the machine learning device 5. The machine learning device 5 includes a control unit 50, a communication unit 51, a learning data storage unit 52, and a trained model storage unit 53.

[0024] The control unit 50 functions as a learning data acquisition unit 500 and a machine learning unit 501. The communication unit 51 is connected to an external device via the network 7 and functions as a communication interface for transmitting and receiving various types of data.

[0025] The learning data acquisition unit 500 is connected to an external device (e.g., the image capturing device 2, the information processing device 6, etc.) via a network 7, for example, and acquires learning data 11 consisting of input data and output data. The learning data 11 is data used as teacher data (training data), verification data, and test data in supervised learning. Furthermore, the output data constituting the learning data 11 is data used as a correct answer label in supervised learning.

[0026] The learning data storage unit 52 is a database that stores a plurality of sets of learning data 11 acquired by the learning data acquisition unit 500. The specific configuration of the database that constitutes the learning data storage unit 52 may be designed as appropriate.

[0027] The machine learning unit 501 performs machine learning using multiple sets of training data 11 stored in the training data storage unit 52. That is, the machine learning unit 501 inputs multiple sets of training data 11 to the training model 12 and causes the training model 12 to learn the correlation between the input data and output data included in the training data 11, thereby generating a trained training model 12. When performing machine learning, the machine learning unit 501 can employ any method, such as online training, batch training, or mini-batch training. Note that the machine learning unit 501 may perform predetermined pre-processing on input data (image data D1) to be input to the training model 12, or may perform predetermined post-processing on output data (safety confirmation determination information D2) output from the training model 12.

[0028] The trained model storage unit 53 is a database that stores the trained learning model 12 (specifically, a set of adjusted weight parameters) generated by the machine learning unit 501. The trained learning model 12 stored in the trained model storage unit 53 is provided to an actual system (e.g., an information processing device 6) via the network 7, a recording medium, or the like. Note that although the training data storage unit 52 and the trained model storage unit 53 are shown as separate storage units in FIG. 2, they may also be configured as a single storage unit.

[0029] 3 is a diagram showing an example of the learning data 11 and the learning model 12. The learning data 11 used for machine learning of the learning model 12 is composed of image data D1 as input data and safety confirmation determination information D2 as output data.

[0030] Image data D1 constituting the input data of the learning data 11 is obtained by photographing a person U using an image capturing device 2. The image data D1 may be either moving image data or still image data. Note that if the image data D1 is moving image data, the input data may include audio data recorded by a microphone or the like when the image data D1 was captured.

[0031] The safety confirmation determination information D2 constituting the output data of the learning data 11 includes a determination result as to whether or not the person U photographed in the image data D1 has correctly performed a safety confirmation action toward a predetermined check point. In this case, the safety confirmation determination information D2 may also include a determination result as to whether or not the head of the person U is facing the check point. The information D2 may include a determination result as to whether or not the safety confirmation operation has been performed correctly for a plurality of confirmation locations in accordance with a predetermined confirmation order.

[0032] In this embodiment, if the safety confirmation action is performed in the order of "right good" and "left good" toward the check points on the right and left, it is determined to be "correct." In contrast, if the safety confirmation action of "right good" is not performed toward the check points on the right, or if the safety confirmation action of "left good" is not performed toward the check points on the left, it is determined to be "incorrect." Examples of person U's actions for which the safety confirmation action is determined to be incorrect include when the head is not facing the check points, or when the pointing direction is not facing the check points. Furthermore, if the image data D1 includes audio data, examples of person U's actions for which the safety confirmation action is determined to be incorrect include when the voice "right good" or "left good" is not uttered.

[0033] The learning data acquisition unit 500 acquires the learning data 11 by referring to the image data D1 and annotation information stored in the image data storage unit 62 of the information processing device 6, or by receiving an input operation from the system administrator of the safety confirmation support system 1 from the information processing device 6. For example, the learning data acquisition unit 500 acquires the learning data 11 by acquiring image data D1 in which the safety confirmation action has been annotated by the system administrator as "correct" or image data D1 in which the safety confirmation action has been annotated by the system administrator as "incorrect."

[0034] The learning model 12 employs, for example, a neural network structure and includes an input layer 120, an intermediate layer 121, and an output layer 122. Synapses (not shown) that connect each neuron are laid between each layer, and each synapse is associated with a weight. A group of weight parameters consisting of the weights of each synapse is adjusted by machine learning.

[0035] The input layer 120 has neurons whose number corresponds to the image data D1 as input data, and each value of the image data D1 is input to each neuron. The output layer 122 has neurons whose number corresponds to the safety confirmation determination information D2 as output data, and the prediction result (inference result) of the safety confirmation determination information D2 for the image data D1 is output as output data.

[0036] The learning model 12 may be, for example, a classification model that classifies actions into a plurality of classes. In this embodiment, as shown in FIG. 3, the learning model 12 outputs a numerical value normalized within a predetermined range (e.g., 0 to 1) as a score (accuracy) for each of four classes: "correct," "'Right-hand' incorrect," "'Left-hand' incorrect," and "'Right-hand' and 'Left-hand' incorrect." The class with the highest score among the scores output for each class is treated as the final safety confirmation operation determination result. Therefore, in the example of FIG. 3, the learning model 12 outputs scores of "0.86," "0.07," "0.05," and "0.02" for the above four classes, respectively, and therefore the safety confirmation determination information D2 is treated as having been determined to be "correct."

[0037] 3, a plurality of data configurations with different conditions, such as the characteristics of the shooting location, the machine learning method, the type of data included in the image data D1, the type of data included in the safety confirmation determination information D2, etc., may be adopted. In this case, the learning data acquisition unit 500 acquires a plurality of types of learning data 11 corresponding to the plurality of data configurations with different conditions, and the machine learning unit 501 performs machine learning using each of the learning data 11.

[0038] For example, if the data included in the safety confirmation determination information D2 is defined as two classes, "correct" and "incorrect," the determination result of the safety confirmation operation included in the safety confirmation determination information D2 represents either "correct" or "incorrect." Furthermore, an additional class, "unable to determine," may be added. In this case, the determination result of the safety confirmation operation included in the safety confirmation determination information D2, "unable to determine," may be treated as being determined to be "incorrect."

[0039] Furthermore, as the learning model 12 that determines whether or not a safety confirmation action including multiple check points has been performed correctly, a single learning model 12 may be used, or multiple learning models 12 may be combined, as shown in Fig. 3. When multiple learning models 12 are used, multiple learning models 12 corresponding to multiple check points may be used, for example, a first learning model that outputs safety confirmation determination information D2 including a determination result of whether or not a "right turn" safety confirmation action has been performed correctly may be combined with a second learning model that outputs safety confirmation determination information D2 including a determination result of whether or not a "left turn" safety confirmation action has been performed correctly.

[0040] (Information processing device 6) Fig. 4 is a block diagram showing an example of the information processing device 6. Fig. 5 is a data configuration diagram showing an example of the image data storage unit 62. Fig. 6 is a functional explanatory diagram showing an example of the information processing device 6. The information processing device 6 includes a control unit 60, a communication unit 61, an image data storage unit 62, and a trained model storage unit 63.

[0041] The control unit 60 functions as an image data acquisition unit 600 and an information processing unit 601. The communication unit 61 is connected to an external device via the network 7 and functions as a communication interface for transmitting and receiving various types of data.

[0042] As shown in FIG. 5, the image data storage unit 62 is, for example, a database that stores image data D1, safety confirmation determination information D2, etc. in association with each other in a table having a plurality of fields in each record.

[0043] In the table of the image data storage unit 62, for example, image data D1 captured during a period from when a person U enters the field of view of the image capture device 2 until when the person U leaves the field of view of the image capture device 2 is stored as one record. At this time, a unique image ID is assigned, and the capture time and capture location are stored. In addition, when safety confirmation determination information D2 for the image data D1 is generated by the information processing unit 601, the safety confirmation determination information D2 is stored. In addition, personal information acquired based on the image data D1 is stored. The personal information includes, for example, a person ID (such as name or affiliation) for identifying the person U captured in the image data D1, a person image (such as face or whole body) cut out from the image data D1, and skeletal data recording the movements of the person U captured in the image data D1 as skeletal (frame) movements. When a system administrator annotates the image data D1 with the determination result of the safety confirmation movement, the annotation information is stored and used as learning data 11.

[0044] The trained model storage unit 63 is a database that stores trained learning models 12 used in the information processing unit 601. The number of trained models 12 stored in the trained model storage unit 63 is not limited to the above example, and multiple trained models with different conditions, such as the characteristics of the shooting location, the machine learning method, the type of data included in the image data D1, the type of data included in the safety confirmation determination information D2, etc., may be stored and used selectively or in parallel. The trained model storage unit 63 may be substituted by a storage unit of an external computer (for example, a server-type computer or a cloud-type computer), in which case In this case, the information processing unit 601 only needs to access the external computer.

[0045] The image data acquisition unit 600 is connected to an external device (e.g., the image capturing device 2) via the communication unit 61 and the network 7, and acquires image data D1 captured by the image capturing device 2. In this case, the image data acquisition unit 600 may acquire (receive) the image data D1 from the image capturing device 2, or may acquire the image data D1 by referring to the image data storage unit 62. Note that the image data acquisition unit 600 may acquire data other than the image data D1 in accordance with the data structure of the input data in the learning model 12, and may further acquire, for example, audio data.

[0046] The information processing unit 601 inputs the image data D1 acquired by the image data acquisition unit 600 as input data into the learning model 12, and generates safety confirmation judgment information D2 including a judgment result as to whether or not the safety confirmation action toward a specified confirmation location was correctly performed as the action of the person U photographed in the image data D1.

[0047] Then, the information processing unit 601 associates the safety confirmation determination information D2 with the image data D1 and stores it in the image data storage unit 62. At this time, the information processing unit 601 assigns a unique image ID and also stores the shooting time and shooting location in the image data storage unit 62. Furthermore, the information processing unit 601 may perform image processing on the image data D1 to obtain person information and store it in the image data storage unit 62.

[0048] Furthermore, when the safety confirmation determination information D2 includes a determination result that the safety confirmation operation was not performed correctly, the information processing unit 601 generates a warning sound from the sound generating device 4 installed on the side of the check location where it was determined that the safety confirmation operation was not performed correctly. In the example of FIG. 6, the class with the highest score, "'Left turn' is incorrect," in the safety confirmation determination information D2 output from the learning model 12 is treated as the final determination result of the safety confirmation operation. Therefore, the information processing unit 601 transmits a sound generation command to the left sound generating device 4L installed on the left check location to generate a warning sound. As a result, upon receiving the sound generation command, the left sound generating device 4L generates a warning sound toward the person U located at the shooting location. This makes it possible to prevent overlooking of check locations.

[0049] The information processing unit 601 may perform predetermined pre-processing on the input data (image data D1) to be input to the learning model 12. For example, when the image data D1 acquired from the image capturing device 2 is continuously captured, the information processing unit 601 may extract only the image data D captured during the period from when the person U enters the angle of view of the image capturing device 2 to when the person U exits the angle of view of the image capturing device 2, and input the extracted image data D to the learning model 12.

[0050] Furthermore, the information processing unit 601 may perform predetermined post-processing on the output data (safety confirmation determination information D2) output from the learning model 12. For example, when multiple learning models 12 are used for a safety confirmation action that includes multiple check points, the safety confirmation determination information D2 output from each learning model 12 may be integrated. For example, by integrating the safety confirmation determination information D2 including the determination result as to whether the safety confirmation action of "right good" was performed correctly and the safety confirmation determination information D2 including the determination result as to whether the safety confirmation action of "left good" was performed correctly, it may be determined whether the series of safety confirmation actions of "right good" and "left good" were performed correctly.

[0051] (Hardware configuration of each device) 7 is a hardware configuration diagram showing an example of a computer 900. Each of the machine learning device 5 and the information processing device 6 is configured by a general-purpose or dedicated computer 900.

[0052] 7, the computer 900 includes, as its main components, a bus 910, a processor 912, a memory 914, an input device 916, an output device 917, a display device 918, a storage device 920, a communication I / F (interface) unit 922, an external device I / F unit 924, an I / O (input / output) device I / F unit 926, and a media input / output unit 928. Note that the above components may be omitted as appropriate depending on the application of the computer 900.

[0053] The processor 912 is composed of one or more arithmetic processing devices (such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), DSP (Digital Signal Processor), GPU (Graphics Processing Unit), or NPU (Neural Processing Unit)), and operates as a control unit that controls the entire computer 900. The memory 914 stores various data and programs 930, and is composed of, for example, a volatile memory (such as a DRAM or SRAM) that functions as a main memory, a non-volatile memory (ROM), a flash memory, etc.

[0054] The input device 916 is composed of, for example, a keyboard, a mouse, a numeric keypad, an electronic pen, etc., and functions as an input unit. The output device 917 is composed of, for example, a sound (audio) output device, a vibration device, etc., and functions as an output unit. The display device 918 is composed of, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, etc., and functions as an output unit. The input device 916 and the display device 918 may be integrated into one device, such as a touch panel display. The storage device 920 is composed of, for example, an HDD, an SSD, etc., and functions as a storage unit. The storage device 920 stores various data necessary for executing the operating system and the program 930.

[0055] The communication I / F unit 922 is connected to a network 940 such as the Internet or an intranet (which may be the same as network 7 in FIG. 1) via a wired or wireless connection and functions as a communication unit that transmits and receives data to and from other computers in accordance with a predetermined communication protocol. The external device I / F unit 924 is connected to an external device 950 such as a camera, printer, scanner, or reader / writer via a wired or wireless connection and functions as a communication unit that transmits and receives data to and from the external device 950 in accordance with a predetermined communication protocol. The I / O device I / F unit 926 is connected to an I / O device 960 such as various sensors and actuators and functions as a communication unit that transmits and receives various signals and data, such as detection signals from sensors and control signals to actuators, to and from the I / O device 960. The media input / output unit 928 is composed of a drive device such as a DVD drive or CD drive, a memory card slot, and a USB connector, and reads and writes data from and to media (non-transitory storage media) 970 such as a DVD, CD, memory card, or USB memory.

[0056] In the computer 900 having the above configuration, the processor 912 loads a program 930 stored in the storage device 920 into the memory 914, executes the program, and controls each unit of the computer 900 via the bus 910. The program 930 may be stored in the memory 914 instead of the storage device 920. The program 930 may be recorded on the medium 970 in an installable file format or an executable file format, and provided to the computer 900 via the media input / output unit 928. The program 930 may be provided to the computer 900 by being downloaded via the network 940 via the communication I / F unit 922. The computer 900 may implement various functions realized by the processor 912 executing the program 930 using hardware such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). It may also be realized by software.

[0057] The computer 900 is an electronic device of any type, such as a desktop computer or a portable computer. The computer 900 may be a client computer, a server computer, a cloud computer, or an embedded computer such as a control panel or a controller (including a microcomputer, a programmable logic controller, or a sequencer).

[0058] (machine learning methods) FIG. 8 is a flowchart showing an example of a machine learning method performed by the machine learning device 5.

[0059] First, in step S100, the learning data acquisition unit 500 acquires a desired number of pieces of learning data 11 as a preliminary preparation for starting machine learning, and stores the acquired learning data 11 in the learning data storage unit 52. The number of pieces of learning data 11 to be prepared here may be set in consideration of the inference accuracy required for the learning model 12 to be finally obtained.

[0060] Next, in step S110, in order to start machine learning, the machine learning unit 501 prepares a pre-learning learning model 12. The pre-learning learning model 12 prepared here is configured with the neural network model exemplified in Fig. 3, and the weight of each synapse is set to an initial value.

[0061] Next, in step S120, the machine learning unit 501 acquires, for example, one set of training data 11 at random from the multiple sets of training data 11 stored in the training data storage unit 52.

[0062] Next, in step S130, the machine learning unit 501 inputs image data D1 (input data) included in a set of learning data 11 to the input layer 120 of the prepared learning model 12 before (or during) learning. As a result, safety confirmation determination information D2 (output data) is output as an inference result from the output layer 122 of the learning model 12, and this output data was generated by the learning model 12 before (or during) learning. Therefore, in the state before (or during) learning, the output data output as an inference result indicates information different from the safety confirmation determination information D2 (correct label) included in the learning data 11.

[0063] Next, in step S140, the machine learning unit 501 performs machine learning by comparing the safety confirmation determination information D2 (correct label) included in the set of learning data 11 acquired in step S120 with the safety confirmation determination information D2 (output data) output as an inference result from the output layer 122 in step S130, and performing a process of adjusting the weight of each synapse (backpropagation).In this way, the machine learning unit 501 causes the learning model 12 to learn the correlation between the input data and the output data.

[0064] Next, in step S150, the machine learning unit 501 determines whether a predetermined learning termination condition has been met, for example, based on the evaluation value of an error function based on the safety confirmation judgment information D2 (correct label) included in the learning data 11 and the safety confirmation judgment information D2 (output data) output as an inference result, or the remaining number of unlearned learning data 11 stored in the learning data memory unit 52.

[0065] In step S150, if the machine learning unit 501 determines that the learning termination condition is not satisfied and that machine learning is to be continued (No in step S150), the process returns to step S120, and the processes of steps S120 to S140 are repeated for the learning model 12 being trained as if it were an untrained learning model. The process is performed multiple times using the training data 11. On the other hand, in step S150, if the machine learning unit 501 determines that the learning end condition is satisfied and that the machine learning is to end (Yes in step S150), the process proceeds to step S160.

[0066] Then, in step S160, the machine learning unit 501 stores the trained learning model 12 (adjusted weight parameter group) generated by adjusting the weights associated with each synapse in the trained model storage unit 53, thereby completing the series of machine learning methods shown in Fig. 8. In the machine learning method, step S100 corresponds to the training data acquisition step, steps S110 to S150 correspond to the machine learning step, and step S160 corresponds to the trained model storage step.

[0067] As described above, the information processing device 6 and safety confirmation support method according to this embodiment can provide a learning model 12 that can predict (infer) safety confirmation judgment information D2 for image data D1 captured by an image capturing device 2 from the image data D1.

[0068] (Safety confirmation support method) 9 is a flowchart showing an example of a safety confirmation support method by the safety confirmation support system 1. In the following, the description is given on the assumption that the image capturing device 2 starts capturing an image in a state where the projection light 30 from the projection device 3 is projected onto the standing position of the person U at the capturing point of the image capturing device 2.

[0069] First, in step S200, the image capturing device 2 generates image data D1 by capturing an image of a person U at a capturing location, and transmits the image data D1 to the information processing device 6.

[0070] Next, in step S210, the image data acquisition unit 600 of the information processing device 6 receives the image data D1 transmitted in step S200, thereby acquiring the image data D1.

[0071] Next, in step S220, the information processing unit 601 inputs the image data D1 acquired in step S210 as input data to the learning model 12, thereby generating safety confirmation determination information D2 for the image data D1.

[0072] Next, in step S230, the information processing unit 601 stores the image data D1 acquired in step S210 and the safety confirmation determination information D2 generated in step S220 in the image data storage unit 62 in association with each other.

[0073] Next, in step S240, when the safety confirmation judgment information D2 includes a judgment result that the safety confirmation operation was not performed correctly, the information processing unit 601 sends a sound generation command to emit a warning sound to the sound generating device 4 installed on the side of the confirmation location where it was judged that the safety confirmation operation was not performed correctly.

[0074] Next, in step S250, upon receiving the sound generation command transmitted in step S240, the sound generation device 4 generates a warning sound toward the person U located at the shooting location. Note that if the safety confirmation determination information D2 includes a determination result that the safety confirmation operation has been performed correctly, the process of generating a warning sound (steps S240, S250) is omitted.

[0075] The above series of operations is performed each time a photograph of the person U is taken as image data D1. Of the above series of operations, step S210 corresponds to the image data acquisition step, and steps S220 to S240 correspond to the information processing step.

[0076] As described above, the safety confirmation support system 1 and the safety confirmation support method according to this embodiment For example, by inputting image data D1 of a person U into the learning model 12, safety confirmation determination information D2 including a determination result as to whether or not the safety confirmation action toward a predetermined check point was correctly performed is generated as the action of the person U photographed in the image data D1. The learning model 12 to which the image data D1 is input is a trained model that has learned the correlation between the image data D1 and the safety confirmation determination information D2 through machine learning, so that it is possible to visually confirm that the safety confirmation action was correctly performed based on the safety confirmation determination information D2 generated by inputting the image data D1 into the learning model 12.

[0077] (Other embodiments) The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention, all of which are included in the technical concept of the present invention.

[0078] In the above embodiment, the machine learning device 5 and the information processing device 6 are configured as separate devices. However, the machine learning device 5 and the information processing device 6 may be configured as a single device. Furthermore, the learning data acquisition unit 500 and the machine learning unit 501 provided in the machine learning device 5 may be incorporated into the image capturing device 2, and the image data acquisition unit 600 and the information processing unit 601 provided in the information processing device 6 may be incorporated into the image capturing device 2.

[0079] In the above embodiment, a case has been described in which a neural network is used as a learning model for realizing machine learning by the machine learning unit 501, but other machine learning models may also be used. Examples of other machine learning models include tree types such as decision trees and regression trees, ensemble learning such as bagging and boosting, and neural network types (including deep learning) such as recurrent neural networks, convolutional neural networks, and LSTM. (including hierarchical clustering, non-hierarchical clustering, k-nearest neighbors, k-means, etc.) Examples include multivariate analysis such as filtering, principal component analysis, factor analysis, and logistic regression, as well as support vector machines.

[0080] (Safety confirmation support program and machine learning program) The present invention can also be provided in the form of a program (safety confirmation support program) for causing the computer 900 to function as each unit included in the safety confirmation support system 1, or a program (safety confirmation support program) for causing the computer 900 to execute each step included in the safety confirmation support method according to the above embodiment. The present invention can also be provided in the form of a program (machine learning program) for causing the computer 900 to function as each unit included in the machine learning device 5, or a program (machine learning program) for causing the computer 900 to execute each step included in the machine learning method.

[0081] (Inference device, inference method and inference program) The present invention can be provided not only in the form of the safety confirmation support system 1 (safety confirmation support method or safety confirmation support program) according to the above embodiment, but also in the form of an inference device (inference method or inference program) used to infer safety confirmation determination information D2. In this case, the inference device (inference method or inference program) can include a memory and a processor, and the processor executes a series of processes. The series of processes includes an image data acquisition process (image data acquisition step) for acquiring image data D1, and an inference process (inference step) for inferring safety confirmation determination information D2 including a determination result as to whether or not a safety confirmation action toward a predetermined check point was correctly performed as a result of the action of person U captured in the image data D1.

[0082] By providing it in the form of an inference device (inference method or inference program), it is possible to implement an information processing device. It is obvious to those skilled in the art that when the inference device (inference method or inference program) infers the safety confirmation determination information D2, the inference method implemented by the information processing unit may be applied using the trained learning model 12 generated by the machine learning device and machine learning method according to the above embodiment. [Explanation of symbols]

[0083] 1...Safety confirmation support system, 2...Image capturing device, 3...Projection device, 4, 4L, 4R...sound generating device, 5...machine learning device, 6...information processing device, 11...Learning data, 12...Learning model, 50...control unit, 51...communication unit, 52...learning data storage unit, 53... learned model storage unit, 60... control unit, 61... communication unit, 62...image data storage unit, 63...trained model storage unit, 500...learning data acquisition unit, 501...machine learning unit, 600...image data acquisition unit, 601...information processing unit

Claims

1. an image data acquisition unit that acquires image data of a person; an information processing unit that inputs the image data acquired by the image data acquisition unit into a learning model to generate safety confirmation determination information including a determination result as to whether or not a safety confirmation action toward a predetermined confirmation point has been correctly performed as the action of the person captured in the image data, The learning model is A trained model that has learned the correlation between the image data and the safety confirmation determination information through machine learning. Safety confirmation support system.

2. The safety confirmation determination information is a determination result as to whether the head of the person is facing the confirmation location, The safety confirmation support system according to claim 1 .

3. The safety confirmation determination information is The result of the determination as to whether the safety confirmation operation has been correctly performed in accordance with a predetermined confirmation order for the plurality of confirmation locations is included. The safety confirmation support system according to claim 1 .

4. an image capturing device that captures an image of the person at a predetermined capturing point and generates the image data; a sound generating device that is installed on the confirmation location side of the photographing location and generates a warning sound from the confirmation location to a person located at the photographing location, The image data acquisition unit acquiring the image data from the image capture device; The information processing unit When the safety confirmation determination information includes a determination result that the safety confirmation operation was not performed correctly, the warning sound is generated from the sound generating device installed on the side of the confirmation location where it was determined that the safety confirmation operation was not performed correctly. The safety confirmation support system according to any one of claims 1 to 3.

5. a projection device that projects a standing position of the person when the safety confirmation action is performed at the shooting location; The safety confirmation support system according to claim 4.

6. An inference device comprising a memory and a processor, The processor: An image data acquisition process for acquiring image data of a person; When the image data is acquired by the image data acquisition process, an inference process is executed to infer safety confirmation determination information including a determination result as to whether or not a safety confirmation action toward a predetermined confirmation point has been correctly performed as the action of the person photographed in the image data. Reasoning device.

7. a learning data acquisition unit that acquires a plurality of sets of learning data each consisting of input data and output data; a machine learning unit that uses the plurality of sets of learning data acquired by the learning data acquisition unit to train a learning model by machine learning to learn a correlation between the input data and the output data; a learned model that stores the learned model in which the correlation is learned by the machine learning unit; a digital storage unit; The input data is The image data is of a person, The output data is safety confirmation determination information including a determination result as to whether or not a safety confirmation action toward a predetermined confirmation point has been correctly performed as a motion of the person captured in the image data; Machine learning device.

8. A safety confirmation support method executed by a computer, comprising: an image data acquisition step of acquiring image data of a person; an information processing step of inputting the image data acquired by the image data acquisition step into a learning model to generate safety confirmation determination information including a determination result as to whether or not a safety confirmation action toward a predetermined confirmation point has been correctly performed as the action of the person captured in the image data, The learning model is A trained model that has learned the correlation between the image data and the safety confirmation determination information through machine learning. Safety confirmation support method.

9. An inference method executed by an inference device having a memory and a processor, The processor: An image data acquisition process for acquiring image data of a person; When the image data is acquired by the image data acquisition process, an inference process is executed to infer safety confirmation determination information including a determination result as to whether or not a safety confirmation action toward a predetermined confirmation point has been correctly performed as the action of the person photographed in the image data. Reasoning method.

10. 1. A computer-implemented machine learning method comprising: a learning data acquisition step of acquiring a plurality of sets of learning data each consisting of input data and output data; a machine learning step of causing a learning model to learn a correlation between the input data and the output data by machine learning using the plurality of sets of learning data acquired by the learning data acquisition step; a learned model storage step of storing the learned model, which has learned the correlation through the machine learning step, in a learned model storage unit; The input data is The image data is of a person, The output data is safety confirmation determination information including a determination result as to whether or not a safety confirmation action toward a predetermined confirmation point has been correctly performed as a motion of the person captured in the image data; Machine learning methods.

Images