Analytical device and program

Abstract

To provide a work series identification device, a work series identification system, a work series identification method, and a program for identifying a work series including a series of individual works performed by a mobile object with high accuracy.SOLUTION: A work series identification device (1) identifies a work series including a series of individual works. The work series identification device (1) includes: acquisition parts (11, 12) for acquiring first sensing information showing positions of a mobile object within a work area in time series, and a plurality of pieces of second sensing information showing ambient environment states at different positions within the work area in time series; and a control part (12) for specifying a sequence of a series of individual works on the basis of the first sensing information, and identifying respective work contents of the series of individual works on the basis of the second sensing information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a work sequence identification device, a work sequence identification system, a work sequence identification method, and a program for identifying a work sequence performed by a moving object.

Background Art

[0002] Patent Document 1 discloses a behavior state estimation system. The behavior state estimation system estimates a person's behavior state from usage information of devices in a building, sensor information detected by sensors in the building, and arrangement relationship information of rooms, devices, and sensors in the building. As a result, even when there are multiple people in the building, the behavior state of each user can be specified without using a camera, and the power consumption by each user can be calculated.

[0003] Patent Document 2 discloses a flow line management system. The flow line management system estimates the start date and time of the flow line of the operation content of a device such as a copy operation or a print operation based on the usage information of the device and the position information of the wireless terminal possessed by the user. As a result, an appropriate flow line for the user using the device can be determined.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure provides a work sequence identification device, a work sequence identification system, a work sequence identification method, and a program for accurately identifying a work sequence including a series of individual operations performed by a moving object.

Means for Solving the Problems

[0006] The work sequence identification device of this disclosure is a work sequence identification device that identifies a work sequence including a series of individual tasks, and comprises: an acquisition unit that acquires first sensing information indicating the position of a moving object in a work area in chronological order, and a plurality of second sensing pieces indicating the ambient environmental conditions at different locations in the work area in chronological order; and a control unit that identifies the order of the series of individual tasks based on the first sensing information and identifies the work content of each of the series of individual tasks based on the second sensing information.

[0007] These general and specific embodiments may be implemented by systems, methods, and computer programs, or combinations thereof. [Effects of the Invention]

[0008] According to the work sequence identification device, work sequence identification system, work sequence identification method, and program of this disclosure, a work sequence including a series of individual tasks performed by a moving object can be accurately identified based on multiple sensing information. The multiple sensing information includes first sensing information that shows the position of the moving object within a work area in chronological order, and multiple second sensing information that shows the ambient environmental conditions at different locations within the work area in chronological order. According to the work sequence identification device, work sequence identification system, work sequence identification method, and program of this disclosure, the order of the series of individual tasks is determined based on the first sensing information, and the content of the individual tasks is identified based on the second sensing information. This enables accurate identification of the work sequence. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram showing the configuration of the work sequence identification system of the first embodiment. [Figure 2] Diagram to explain the camera position [Figure 3] Diagram to explain the location of the nearby sensor [Figure 4] A diagram illustrating the types of sensors included in the proximity sensor. [Figure 5]An example of sensor position information indicating the position of a proximity sensor [Figure 6] An example of classification information indicating the range of a work location [Figure 7] An example of start / end information indicating the start and end positions of work for each work type [Figure 8] An example of movement line data obtained from camera video data [Figure 9] An example of ambient environment data generated by a proximity sensor [Figure 10] A diagram for explaining an example of identifying the work content of individual work in the first embodiment [Figure 11] A diagram for explaining an example of work sequence information [Figure 12] A diagram showing an example of the display of a work sequence [Figure 13] A diagram showing camera video display in the display of a work sequence [Figure 14] A diagram showing another example of the display of a work sequence [Figure 15] A flowchart for explaining the operation of a work sequence identification device in the first embodiment [Figure 16] A flowchart showing the details of the work sequence information generation process (step S4) in FIG. 15 [Figure 17] A block diagram showing the configuration of a work sequence identification system in the second embodiment [Figure 18] A diagram for explaining an example of identifying the work content of individual work in the second embodiment [Figure 19] A flowchart for explaining the calculation of the operation rate in the third embodiment

Embodiments for Carrying Out the Invention

[0010] (Knowledge on which the present disclosure is based) Conventionally, in a work site where employees work, such as a kitchen or a factory, in order to review time-consuming work and improve work efficiency such as the variation in time taken by each employee, the time taken for each individual work in a work sequence including a series of individual works has been measured by an improvement person intervening. For example, in the food service industry, when aiming to improve the efficiency of cooking work in a kitchen, the improvement person arranges a camera in the kitchen and, by referring to the camera image and the like, identifies the work content of each individual work in the work sequence from receiving an order to providing the cooked dish, and measures the time taken for each individual work with a stopwatch or the like.

[0011] In recent years, there has been an attempt to cyber-physical system (CPS)ize by reproducing and analyzing the process of the on-site space in a virtual space and proposing improvement plans for the site. In CPSization, in order to efficiently measure work time, for example, it is required to automate the calculation of work time. For this purpose, it is desirable to identify the work sequence performed by employees only by equipment without human intervention.

[0012] This embodiment provides a work sequence identification device that accurately identifies a work sequence including a series of individual works performed by a moving object such as a person. Specifically, the work sequence identification device identifies the work sequence based on wide-area sensing information generated by a remote sensor such as a camera and proximity sensing information generated by a proximity sensor that detects vibration and electromagnetic waves.

[0013] (First Embodiment) In this embodiment, a case where the moving object is a person and a series of individual works performed by the person are works in a kitchen will be described as an example of identifying a work sequence including a series of individual works. The series of individual works in the kitchen includes, for example, a plurality of individual works from receiving an order to providing the cooked dish. The plurality of individual works are, for example, receiving an order, opening and closing the refrigerator, opening and closing the dish cabinet, serving the ingredients, and providing the cooked dish.

[0014] 1. Configuration of Work Sequence Identification System Figure 1 shows the electrical configuration of the work sequence identification system 100 of this embodiment. The work sequence identification system 100 includes a work sequence identification device 1, a camera 2, and a plurality of proximity sensors 3. In this embodiment, the work sequence identification device 1, the camera 2, and the plurality of proximity sensors 3 are installed in a restaurant that serves food. The work sequence identification device 1 and the camera 2 are connected, for example, by a wired connection. The work sequence identification device 1 is connected to the plurality of proximity sensors 3, for example, via a wireless router. The work sequence identification device 1 identifies a work sequence, which includes a series of individual tasks performed by an employee in the kitchen, based on video data generated by the camera 2 and ambient environment data generated by the plurality of proximity sensors 3.

[0015] 1.1 Configuration of the work sequence identification device The work sequence identification device 1 is, for example, a personal computer or tablet terminal installed in a store, or any other type of information processing device. The work sequence identification device 1 comprises a communication unit 11, a control unit 12, a storage unit 13, an input unit 14, a display unit 15, and a bus 16.

[0016] The communication unit 11 includes a circuit that communicates with external devices in accordance with a predetermined communication standard. The predetermined communication standard is, for example, LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), USB, and HDMI (registered trademark). The communication unit 11 acquires video data from the camera 2 and ambient environment data from each of the multiple nearby sensors 3.

[0017] The control unit 12 can be implemented using semiconductor elements or the like. For example, the control unit 12 can be composed of a microcontroller, CPU, MPU, GPU, DSP, FPGA, or ASIC. The functions of the control unit 12 may be implemented using hardware alone, or by combining hardware and software. The control unit 12 performs predetermined functions by reading data and programs stored in the memory unit 13 and performing various arithmetic operations.

[0018] The storage unit 13 is a storage medium that stores the programs and data necessary to realize the functions of the work sequence identification device 1. The storage unit 13 can be implemented, for example, by a hard disk (HDD), SSD, RAM, DRAM, ferroelectric memory, flash memory, magnetic disk, or a combination thereof.

[0019] The communication unit 11 corresponds to an acquisition unit that acquires video data and ambient environment data from the camera 2 and the nearby sensor 3, respectively. The video data and ambient environment data acquired via the communication unit 11 are stored in the storage unit 13. The control unit 12 corresponds to an acquisition unit that reads out the video data and ambient environment data stored in the storage unit 13.

[0020] The input unit 14 is a user interface for inputting various operations by the user. The input unit 14 can be implemented as a touch panel, keyboard, buttons, switches, or a combination thereof.

[0021] The display unit 15 is, for example, a liquid crystal display or an organic EL display. The display unit 15 displays, for example, the work sequence identified by the control unit 12.

[0022] Bus 16 is a signal line that electrically connects the communication unit 11, the control unit 12, the storage unit 13, the input unit 14, and the display unit 15.

[0023] 1.2 Camera Configuration Figure 2 shows an example of a location where camera 2 is placed. Camera 2 is, for example, an all-around camera installed on the ceiling of a kitchen in a store. Camera 2 is equipped with an image sensor such as a CCD image sensor, a CMOS image sensor, or an NMOS image sensor. Camera 2 captures images of the kitchen and generates video data. Camera 2 is an example of a remote sensor that detects the position of moving objects within a work area. The video data generated by camera 2 is an example of first sensing information, in other words, wide-area sensing information, that shows the position of moving objects within the work area in chronological order. The video data includes time information indicating the time of capture.

[0024] 1.3 Configuration of the Nearby Sensor Figure 3 shows an example of the locations where multiple proximity sensors 3 are placed. The multiple proximity sensors 3 are installed at different locations within the kitchen. The control unit 12 divides the kitchen into multiple work areas, in the example of Figure 3, into work areas L1 to L10 and manages them. In the example of Figure 3, the shape of each work area is rectangular, but the way the work areas are divided is arbitrary and they do not have to be rectangular. The number of work areas to be divided can be appropriately determined according to the size of the entire work area. Each proximity sensor 3 is installed, for example, in one of the multiple work areas L1 to L10. Specifically, for example, each proximity sensor 3 is installed on the door or inside of a refrigerator, on the door or inside a cupboard, near a microwave oven, near the serving counter, etc. In this specification, when work areas L1 to L10 are not specifically distinguished, they are collectively referred to as work area L. The multiple proximity sensors 3 detect the surrounding environmental state at their respective locations and generate surrounding environmental data that shows the surrounding environmental state in chronological order. Ambient environment data is a second type of sensing information, or in other words, an example of proximity sensing information, that shows the state of the surrounding environment at different locations within the work area in a time series.

[0025] Figure 4 shows an example of the types of sensors included in each proximity sensor 3. In this embodiment, the proximity sensor 3 is a multimodal sensor in which multiple sensors are packaged in one device. For example, each proximity sensor 3 contains two or more of the following in this embodiment: a human presence sensor 31, a sound sensor 32, a vibration sensor 33, an electromagnetic wave sensor 34, an acceleration sensor 35, and a temperature sensor 36, all of which are included in one housing. The proximity sensor 3 may also include one or more of the following: a geomagnetic sensor, a humidity sensor, an illuminance sensor, and a barometric pressure sensor. Alternatively, the proximity sensor 3 may contain only one sensor.

[0026] 2. Data used in the work sequence identification system Figure 5 shows an example of sensor position information indicating the position and sensitivity of each of the multiple nearby sensors 3. The sensor position information 51 is pre-stored in, for example, the memory unit 13. The sensor position information 51 includes an ID, which is the identification information of the nearby sensor 3, and x,y coordinate values ​​indicating the position of the nearby sensor 3. The sensor position information 51 further includes a value of the radius r centered on the nearby sensor 3, which indicates the sensitivity of the nearby sensor 3.

[0027] Figure 6 shows an example of partitioning information indicating the location of the work area L. The partitioning information 52 is pre-stored in, for example, the memory unit 13. In this embodiment, as shown in Figure 3, the area within the kitchen is divided into multiple rectangular work areas L, so the partitioning information 52 includes x,y coordinate values ​​x1,y1,x2,y2 of the diagonal positions representing the range of each work area L.

[0028] Figure 7 shows an example of start and end information indicating the start and end positions for each type of work. The start and end information 53 is pre-stored in, for example, the memory unit 13. Work types A, B, and C are, for example, the preparation process performed during opening preparations, the cooking process performed during business hours, and the cleanup process performed during closing preparations. The start and end positions are indicated by work locations L1 to L10, respectively. Alternatively, the start and end positions may be indicated by x,y coordinates within the kitchen instead of work locations L1 to L10.

[0029] Figure 8 shows an example of movement path data. The control unit 12 of the work sequence identification device 1 identifies moving objects such as people from the video data of camera 2 through image analysis and generates movement path data 54 that indicates the position of the moving objects. If multiple people are visible in the video data at the same time, the control unit 12 generates movement path data 54 for each person, for example. The movement path data 54 represents a movement path M1, for example, as shown in Figure 3. Specifically, the movement path data 54 in this embodiment shows the position of each person at each time point in time using x,y coordinates.

[0030] Figure 9 shows an example of ambient environment data generated by each nearby sensor 3. The ambient environment data 55 includes detected values ​​d1 to dn at each time point. The detected values ​​d1 to dn are values ​​detected by the human presence sensor 31, sound sensor 32, vibration sensor 33, electromagnetic wave sensor 34, acceleration sensor 35, and temperature sensor 36. That is, the detected values ​​d1 to dn correspond to either infrared radiation, sound, vibration, electromagnetic waves, acceleration, or temperature. Depending on the type of sensor included in the nearby sensor 3, the ambient environment data 55 may also include detected values ​​for geomagnetic field, humidity, illuminance, and atmospheric pressure.

[0031] Figure 10 shows an example of a method for identifying the work content of individual tasks. The control unit 12 uses the individual task identifier 120 to identify the work content of individual tasks. The program and parameters for constructing the individual task identifier 120 are stored, for example, in the storage unit 13.

[0032] The individual task classifier 120 includes a model trained to identify the content of individual tasks using machine learning such as deep learning. For example, the individual task classifier 120 is composed of a recurrent neural network. The recurrent neural network has a multilayer structure including, for example, an input layer N1, hidden layers N2, N3, N4, and an output layer N5. Note that the number of hidden layers is not limited to three. The hidden layers include one or more layers. The individual task classifier 120 may also employ other types of machine learning algorithms.

[0033] The individual task identifier 120, for example, when it receives detection values ​​d1 to dn of ambient environment data 55 for a predetermined time period from the input layer N1, outputs probabilities P(W1) to P(Wm) from the output layer N5 that indicate which task content is being represented. The predetermined time period is, for example, 10 seconds. Each of the task content W1 to Wm corresponds to, for example, in the case of a cooking process, taking an order, opening and closing the refrigerator, opening and closing the cupboard, using the microwave oven, plating, and serving the food. In this specification, when task content W1 to Wm are not specifically distinguished, they are collectively referred to as task content W.

[0034] The recurrent neural network is trained based on training data, which is pre-associated with ambient environmental data for training and correct labels indicating the work content corresponding to the ambient environmental data for training. In other words, the weighting coefficients of each layer of the recurrent neural network that constitutes the individual work classifier 120 are set by training using the training data. For example, the weighting coefficients between the nodes of each layer are set so that the probability corresponding to the work content of the correct label is maximized among the probabilities P(W1) to P(Wm) obtained by inputting detection values ​​d1 to dn for 10 seconds into the individual work classifier 120. The training of the individual work classifier 120 may be performed by the work sequence identification device 1 or by another information processing device.

[0035] When the control unit 12 identifies a task from the ambient environment data 55, it inputs the detected values ​​d1 to dn for a predetermined time period into the trained individual task classifier 120 to calculate the probability P(W1) to P(Wm) of the task. For example, the control unit 12 determines that the task indicated by the detected values ​​d1 to dn for a predetermined time period input to the input layer N1 is the task with the largest value among the probabilities P(W1) to P(Wm) obtained from the output layer N5.

[0036] Figure 11 is a schematic diagram showing an example of work sequence information generated by the control unit 12. The work sequence information 56 represents a graph showing information including, for example, work location, work content, and work time in chronological order. The work location L described in the work sequence information 56 is one of the work locations L1 to L10 shown in Figure 3. The work content W described in the work sequence information 56 is the result of identification using the individual work identifier 120 shown in Figure 10, and is one of the work content W1 to Wm. The work time T described in the work sequence information 56 indicates the time T1, T2, etc., when the individual work of work content W was performed at work location L.

[0037] Figures 12 to 14 show examples of how the display unit 15 displays a work series based on work series information 56. In the display example in Figure 12, the work series is shown, that is, the order, location, and content of a series of individual tasks, along with the minimum, average, maximum, standard deviation, and ranking of the work time for each individual task. In the display example in Figure 13, camera footage is displayed. For example, when the display unit 15 is displaying the screen shown in Figure 12, and the user selects an individual task from the displayed series of individual tasks and a representative value (one of the minimum, average, maximum, standard deviation, and ranking) for the work time of that individual task via the input unit 14, the camera footage of the individual task corresponding to the selected representative value is displayed, as shown in Figure 13. The ranking can be set to the average, maximum, or standard deviation of the work time. This allows, for example, the user to check the corresponding camera footage for tasks that are taking a long time or individual tasks with large variations in work time, in ranking order. In the display example in Figure 14, the work series is shown, that is, the order, location, and content of a series of individual tasks, along with the work time for each individual task performed by a certain person.

[0038] 3. Operation of the work sequence identification device 3.1 Overall Operation The operation of the work sequence identification device 1 with the above configuration will now be explained. Figure 15 shows the operation of the control unit 12 of the work sequence identification device 1.

[0039] The control unit 12 acquires video data from camera 2 (S1). For example, the work sequence identification device 1 acquires the video data generated by camera 2 via the communication unit 11 and stores it in the storage unit 13. In step S1, the control unit 12 reads the video data stored in the storage unit 13. For example, the control unit 12 reads out one day's worth of video data.

[0040] The control unit 12 acquires information on the start and end positions according to the type of work (S2). For example, when the user specifies a type of work via the input unit 14, the control unit 12 reads the start and end positions corresponding to the type of work specified by the user from the start and end information 53.

[0041] The control unit 12 detects the movement path from the starting position to the ending position from the video data (S3). For example, the control unit 12 performs image analysis of the video data to identify a moving person, detects the person's movement path from the starting position to the ending position, and generates movement path data 54.

[0042] The control unit 12 generates work sequence information 56 indicating a series of individual tasks along the movement path based on the movement path data 54 (S4).

[0043] The control unit 12 determines whether the detection of movement paths in the video has been completed (S5). For example, if there is any video data remaining from step S1 that has not been attempted for person identification and movement path detection in step S3, the process returns to step S3. As a result, the control unit 12 generates work sequence information 56 for each movement path detected in step S3. Once the attempts for person identification and movement path detection have been completed for all video data acquired in step S1, the process proceeds to step S6.

[0044] Based on the work series information 56 generated in step S4, the control unit 12 calculates the minimum, average, maximum, standard deviation, and ranking of the work time for each individual task (S6). In this embodiment, the control unit 12 calculates all of the minimum, average, and maximum values, standard deviation, and ranking, but it may calculate one or more of them.

[0045] The control unit 12 displays the work location, work content, and work time on the display unit 15 (S7). For example, as shown in Figure 12, the display unit 15 displays the work content of the identified individual work, the order of each individual work, and the minimum, average, maximum, standard deviation, and ranking of the work time for each individual work near the specified work location. While the display unit 15 is displaying the screen shown in Figure 12, the control unit 12 may accept a selection from the user via the input unit 14 for any individual work and representative values ​​of the work time (any of the minimum, average, maximum, standard deviation, and ranking). For example, the user may select the individual work and representative values ​​of the work time using a touch panel. In this case, the control unit 12 may select the individual work and representative values ​​of the work time according to the user's selection and display the camera image of the individual work corresponding to the selected representative values ​​on the display unit 15, as shown in Figure 13. Alternatively, the control unit 12 may select any individual work and representative values ​​of the work time instead of the user's selection. For example, the control unit 12 may sequentially display camera images corresponding to the minimum, average, maximum, standard deviation, or ranking of each individual task on the display unit 15. In place of or in addition to the summary display shown in Figure 12, the display unit 15 may display the work time for a particular person, as shown in Figure 14. For example, the control unit 12 may display the work time for each individual task, along with the work content and order, for the person with the maximum and minimum work time for the entire work series, near the specified work location.

[0046] 3.2 Operation of generating work sequence information Figure 16 shows the operation of generating the work sequence information 56, i.e., the details of step S4 in Figure 15.

[0047] The control unit 12 identifies a work location based on the movement path data 54 and the classification information 52 (S401). The control unit 12 selects a nearby sensor 3 to detect the surrounding environment conditions at the identified work location by referring to the sensor position information 51 (S402). This selects the surrounding environment data 55 to be used to identify the work content from among the multiple surrounding environment data 55 stored in the storage unit 13. In this embodiment, the control unit 12 selects all nearby sensors 3 within a predetermined range from the identified work location. In another example, the control unit 12 may select the nearest nearby sensor 3 to the movement path.

[0048] The control unit 12 acquires the ambient environment data 55 generated by the selected nearby sensor 3 from the storage unit 13 (S403).

[0049] The control unit 12 determines whether there are two or more nearby sensors 3 selected in step S402 (S404).

[0050] If there are two or more nearby sensors 3 selected in step S402 (Yes in S404), the control unit 12 determines one nearby sensor 3 from among those two or more nearby sensors 3 based on the ambient environment data 55 acquired in step S403 (S405). For example, the control unit 12 associates the movement data 54 and the ambient environment data 55 based on the time information contained in each. The control unit 12 may select a nearby sensor 3 whose detection value from the human presence sensor 31 indicates that a person is closer when a person is present at the work location identified in step S401. It may also select a nearby sensor 3 whose detection value from the electromagnetic wave sensor 34 is higher. It may also select a nearby sensor 3 whose detection value from the sound sensor 32 is higher. The control unit 12 may also change which of the detection values ​​d1 to dn included in the ambient environment data 55 is referenced depending on the work location.

[0051] If there is only one nearby sensor 3 selected in step S402 (No in S404), step S405 is omitted and the process proceeds to step S406.

[0052] The control unit 12 identifies the work content of an individual task based on the ambient environment data 55 generated by the determined proximity sensor 3 (S406). For example, the control unit 12 associates the movement data 54 and the ambient environment data 55 based on the time information contained in each. The control unit 12 extracts the detected values ​​d1 to dn from the ambient environment data 55 while a person is in the work location identified in step S401. The control unit 12 inputs the detected values ​​d1 to dn for a predetermined time period from the extracted time period to the individual task identifier 120. For example, if the detected values ​​d1 to dn for one minute are extracted, the detected values ​​d1 to dn for 10 seconds within that period may be input sequentially to the individual task identifier 120. The control unit 12 may also input the detected values ​​d1 to dn for a predetermined time period from the moment a person enters the radius r range indicating the sensitivity of the proximity sensor 3, based on the movement data 54 and the sensor position information 51, to the individual task identifier 120.

[0053] The control unit 12 calculates the work time required for individual tasks (S407). For example, the control unit 12 calculates the work time at the work location identified in step S401 based on the time information included in the movement data 54. The work time may also be calculated based on the time information indicated by the surrounding environment data 55. For example, the control unit 12 may calculate the time during which it is determined that the same individual task is being performed based on the output result of the individual task identifier 120 in step S406.

[0054] The control unit 12 adds the work location identified in step S401, the work content identified in step S406, and the work time calculated in step S407 to the work series information 56 (S408).

[0055] The control unit 12 executes steps S401 to S408 in order from the starting position of the movement data 54. The control unit 12 determines whether the work location identified in step S401 includes the end position of the movement and whether the movement has ended (S409). If the work location identified in step S401 does not include the end position of the movement, the process returns to step S401. As a result, steps S401 to S408 are executed for the next work location in the movement data 54. If the work location identified in step S401 includes the end position of the movement and the movement has ended, the work sequence information generation process shown in Figure 16 is terminated.

[0056] 4. Effects and Supplementary Information The work sequence identification system 100 of this embodiment comprises a work sequence identification device 1, a camera 2, and a plurality of proximity sensors 3. The camera 2 photographs the work area and generates video data showing the position of a moving object within the work area in chronological order. The plurality of proximity sensors 3 are positioned at different locations within the work area and generate ambient environment data 55 showing the ambient environment conditions at their respective positions in chronological order. The ambient environment conditions include one or more of the following: sound, vibration, infrared radiation, electromagnetic waves, geomagnetic field, acceleration, temperature, humidity, illuminance, atmospheric pressure, and carbon dioxide concentration. The work sequence identification device 1 identifies a work sequence, which includes a series of individual tasks performed by a moving object, based on the video data and the ambient environment data 55.

[0057] Specifically, the work sequence identification device 1 comprises an acquisition unit and a control unit 12. The acquisition unit is either a communication unit 11 or the control unit 12. The acquisition unit acquires video data and ambient environment data 55. The control unit 12 identifies the work sequence by determining the order of a series of individual tasks based on the video data and identifying the content of each individual task based on the ambient environment data.

[0058] The video data generated by the ceiling-mounted camera 2 can detect the movement of people. However, it is difficult to identify the work being done from this video data. The ambient environment data 55 generated by the proximity sensor 3 can identify the work being done. However, it is difficult to identify people and detect their movement from this ambient environment data 55. In this embodiment, by using both the video data and the ambient environment data 55, both the detection of people's movement and the identification of work being done can be achieved. Therefore, the work sequence can be identified with high accuracy.

[0059] The acquisition unit acquires ambient environment data 55 from multiple proximity sensors 3 positioned at different locations within the work area. The control unit 12 selects ambient environment data to be used to identify the work content from among the multiple ambient environment data 55 based on the position of the moving object and the positions of the multiple proximity sensors 3. If the control unit 12 has selected multiple ambient environment data to be used to identify the work content based on the position of the moving object and the positions of the multiple proximity sensors 3, it determines which ambient environment data to actually use based on the selected multiple ambient environment data. This enables accurate identification of the work content.

[0060] The control unit 12 divides the work area into multiple work locations and manages them. Based on video data, it identifies the work location where a moving object is located from among the multiple work locations, and generates work sequence information 56 by associating the work location with the work content. Since the generation of work sequence information 56 can be automated, there is no need to generate it manually. Therefore, work sequence information 56 can be generated efficiently. The work sequence information 56 generated in this way is useful for analyzing the work and planning improvements such as equipment placement or work flow.

[0061] The control unit 12 detects the movement path of moving objects from a start position to an end position within the work area, according to the type of work, based on video data. It repeatedly identifies work locations and work content along the detected movement path to generate work sequence information 56. This makes it possible to generate work sequence information 56 along the movement path according to the type of work.

[0062] The video data includes time information indicating the time of capture, and the ambient environment data 55 includes time information indicating the time of sensing. The ambient environment data 55 and the movement path data 54 generated from the video data are synchronized based on the time information. The control unit 12 calculates the work time based on the time information. This makes it possible to automatically measure the work time for each individual task included in the work sequence.

[0063] The control unit 12 calculates at least one of the following for the same individual task: minimum, average, maximum, standard deviation, and ranking of the work time. The calculated values ​​are useful for analyzing the work and planning improvements such as equipment placement or workflow.

[0064] The work sequence identification device 1 further includes a display unit 15 that displays the sequence, work location, and work content of a series of individual tasks, as well as at least one of the minimum, average, maximum, standard deviation, and ranking of the time taken for each individual task. This allows the user to visually confirm the work sequence and the work time.

[0065] In this embodiment, the work area is the area within the kitchen, and the series of individual tasks includes tasks related to cooking. The work sequence information 56 related to the tasks within the kitchen is useful for analyzing the tasks within the kitchen and planning improvements such as the arrangement of equipment or the flow of work.

[0066] (Second Embodiment) In the first embodiment described above, in step S406, the control unit 12 identified the work content of the individual task based on the ambient environment data 55. In this embodiment, in addition to the ambient environment data 55, the control unit 12 uses POS (Point of Sales) information indicating the purchased products to identify the work content of the individual task.

[0067] Figure 17 shows the electrical configuration of the work sequence identification system 100 of this embodiment. The work sequence identification system 100 of this embodiment further includes a work sequence identification device 1, a camera 2, and a plurality of nearby sensors 3, as well as a POS terminal 4. The POS terminal 4 generates POS information. The POS information includes the time at the time of sale, product name, quantity, and amount. The work sequence identification device 1 acquires the POS information generated by the POS terminal 4 via the communication unit 11. The communication unit 11 corresponds to an acquisition unit that acquires POS information from the POS terminal 4. The acquired POS information is stored in the storage unit 13. The control unit 12 corresponds to an acquisition unit that reads the POS information from the storage unit 13.

[0068] Figure 18 shows an example of identifying the content of individual tasks using POS information in this embodiment. In step S406, the control unit 12 sets coefficients q1 to qm used to identify the work content based on the purchased products indicated by the POS information. For example, coefficients q1 to qm for each of the work contents W1 to Wm are predetermined for each product, and information indicating the coefficients q1 to qm for each product is stored in the storage unit 13. Before step S406, the control unit 12 reads the POS information from the storage unit 13, and in step S406, reads the coefficients q1 to qm corresponding to the purchased products indicated by the POS information from the storage unit 13. The control unit 12 multiplies the probabilities P(W1) to P(Wm) of each work content calculated by the individual task identifier 120 by the coefficients q1 to qm corresponding to the purchased products. That is, the control unit 12 calculates "P(W) × q" for each of the m individual tasks output from the individual task identifier 120. The control unit 12 determines that the work content W that maximizes the calculated value of "P(W) × q" is the work content of the individual work corresponding to the detected values ​​d1 to dn input to the individual work identifier 120. The control unit 12 writes the work content W identified in this way to the work sequence information 56.

[0069] As described above, the acquisition unit of this embodiment acquires POS information indicating the purchased product in addition to ambient environment data 55. The control unit 12 identifies the work content based on the POS information and ambient environment data 55. This enables accurate identification of the work content.

[0070] In this embodiment, the work content was determined by calculating the product of the probability P(W) and coefficient q for each of the m individual work output from the individual work classifier 120. However, the coefficient q may be pre-trained as training data for the individual work classifier 120. In this case, the individual work classifier 120 is input with the detected value d and the coefficient q, and the work content with the largest value among the probabilities P(W1) to P(Wm) obtained from the output is determined to be the work content corresponding to the input detected value d.

[0071] (Third embodiment) In the first and second embodiments, examples were described in which the work sequence identification system 100 identifies a work sequence including a series of individual tasks and generates work sequence information 56 indicating the work sequence. In this embodiment, the work sequence identification system 100 calculates the utilization rate of a moving object performing a series of individual tasks. Specifically, the control unit 12 of the work sequence identification device 1 calculates the utilization rate by calculating the time during which the moving object is not operating along the movement path.

[0072] Figure 19 is a flowchart showing the operation of the control unit 12 of the work sequence identification device 1 in the third embodiment for calculating the utilization rate. The control unit 12 calculates the utilization rate shown in Figure 19 after generating the work sequence information 56 shown in Figures 15 and 16. The calculation of the utilization rate shown in Figure 19 may be performed in parallel with the generation of the work sequence information 56 shown in Figures 15 and 16. Steps S11 to S13 in Figure 19 are the same as steps S1 to S3 in Figure 15 of the first embodiment. Steps S16 to S18 in Figure 19 correspond to steps S401 to S405 in Figure 16 of the first embodiment.

[0073] The control unit 12 acquires video data from camera 2 (S11). The control unit 12 acquires information on the start and end positions according to the type of work (S12). The control unit 12 detects the movement path from the start position to the end position from the video data (S13).

[0074] The control unit 12 calculates the length of the first period based on the time from the start to the end of the movement path (S14). For example, the control unit 12 calculates the length of the first period based on the time information contained in the movement path data 54.

[0075] The control unit 12 calculates the length of the second period (S15) as the time during which the amount of movement of the movement path is less than or equal to a first threshold, within the period from the start position to the end position of the movement path detected in step S13. For example, based on the movement path data 54, the control unit 12 calculates the time during which the sum of the changes in the x and y coordinates is less than or equal to a first threshold, based on the time information contained in the movement path data 54.

[0076] The control unit 12 identifies the work location within the second period based on the movement data 54 and the classification information 52 (S16). The control unit 12 refers to the sensor position information 51 and selects a nearby sensor 3 to detect the surrounding environment conditions at the identified work location (S17). The control unit 12 obtains the surrounding environment data 55 generated by the nearby sensor 3 within the second period from the storage unit 13 (S18). If there are multiple nearby sensors 3 selected in step S17, the control unit 12 determines one nearby sensor 3 to be used to identify the work content based on the surrounding environment data 55 generated by the selected nearby sensors 3.

[0077] The control unit 12 calculates the length of the third period as the time during which the change in the detected value in the ambient environment data 55 within the second period is less than or equal to the second threshold (S19). For example, the control unit 12 calculates the time during which the detected value of electromagnetic waves included in the ambient environment data 55 within the second period is less than or equal to the second threshold, based on the time information included in the ambient environment data 55. The control unit 12 may change which of the detected values ​​d1 to dn included in the ambient environment data 55 is used, depending on the work location identified in step S16. The control unit 12 may also compare all of the detected values ​​d1 to dn included in the ambient environment data 55 with the corresponding second threshold. If multiple second periods are calculated within the range from the start position to the end position of the movement path in step S15, the control unit 12 performs the processing shown in steps S16 to S19 for each second period and calculates the length of the third period for each.

[0078] The third period is the time when the amount of movement along the movement path is below the first threshold and the amount of change in the detected value in the surrounding environment data 55 is below the second threshold. In other words, the third period represents the non-operational time of the moving object.

[0079] The control unit 12 calculates the utilization rate (S20). Specifically, the control unit 12 calculates the utilization rate using the formula: "Utilization rate = (Sum of first period - sum of third period) / first period". Alternatively, the control unit 12 may calculate the non-utilization rate using the formula: "Non-utilization rate = Sum of third period / first period".

[0080] The control unit 12 may add the calculated operating rate to the work series information 56 or display it on the display unit 15. The control unit 12 may add the non-working time, which is the length of the third period, to the work series information 56 or display it on the display unit 15. For example, the control unit 12 may associate the work location identified in step S16 with the non-working time and add it to the work series information 56. The display unit 15 may display the non-operating time near the work area.

[0081] In this way, the control unit 12 calculates the operational rate of the moving object based on the proportion of the time within the total time of the detected movement path during which the amount of change in the position of the moving object is below a first threshold and the amount of change in the surrounding environmental state is below a second threshold. The data showing the operational rate is useful for analyzing the work and planning improvements such as equipment placement or work flow.

[0082] (Other embodiments) As described above, the first to third embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited thereto and can be applied to embodiments that have been modified, replaced, added, or omitted as appropriate. Therefore, other embodiments will be illustrated below.

[0083] The first to third embodiments described above illustrate an example of identifying a work sequence that includes a series of individual tasks performed by a person in a kitchen. However, the scope of work sequence identification is not limited to tasks performed by a person in a kitchen. For example, the scope of work sequence identification is not limited to tasks performed in a kitchen, but may also include tasks performed in a factory. The scope of work sequence identification is not limited to tasks performed by a person, but may also include tasks performed by machines such as forklifts or robots. The work sequence identification device 1, camera 2, and multiple proximity sensors 3 may be installed in a factory. Camera 2 may photograph the range in which a forklift moves within the factory. The work sequence identification device 1 may use the proximity sensors 3 to detect the operation of the forklift. The proximity sensors 3 can be used to detect human actions, machine actions, and the on-site environment. Human actions include, for example, product assembly, picking, and bento box preparation. Machine actions include line operations, forklift operations, and the status of cooking equipment. The on-site environment includes noise, temperature, humidity, carbon dioxide concentration, and illuminance.

[0084] Furthermore, the first to third embodiments described above illustrate an example of identifying a work sequence including a series of individual tasks within a kitchen in a store. However, its application is not limited to commercial use such as stores, but may also be to kitchens or rooms in homes. For example, in a home kitchen, the work sequence identification device 1, camera 2, and multiple proximity sensors 3 may be placed in the kitchen. The proximity sensors 3 can be used to detect the actions of the cook, the actions of cooking utensils, and the kitchen environment. The actions of the cook include taking out and putting away food, cooking food, and taking out and putting away dishes. The actions of cooking utensils include opening and closing food shelves, the status of cooking utensils, and opening and closing storage shelves. The kitchen environment includes noise, temperature, humidity, carbon dioxide concentration, and illuminance. This allows for the identification of a work sequence of cooking tasks in the kitchen.

[0085] In the above embodiment, camera 2 was used as an example of a remote sensor, but the remote sensor is not limited to camera 2. Any sensor capable of detecting the position of a moving object is acceptable as a remote sensor. In the above embodiment, the work sequence identification device 1 generated movement path data 54 by performing image analysis on video data, but the work sequence identification device 1 may also acquire movement path data 54 from another device equipped with camera 2 or from another remote sensor that detects the position of a moving object.

[0086] Furthermore, in the above embodiment, the start and end positions according to the type of work were detected from the start / end information 53, but the start and end positions may also be detected from the video data of camera 2. Specifically, the movement path position and movement difference information obtained from camera 2 are used, and if the movement path is included in a specific range and the movement difference information exceeds a predetermined threshold, that position is detected as the start or end position of the work. This is useful when the start and / or end of work occurs in a location other than the normally expected location. For example, when serving food, employees reach out their hands, so the movement difference value at that time will be a large value. Therefore, even if it is a location other than the normal food serving location near the counter, if the movement difference value is above the threshold, it can be detected as the end position of the work.

[0087] (Summary of the embodiment) (1) The work sequence identification device of the present disclosure is a work sequence identification device that identifies a work sequence including a series of individual tasks, and comprises: an acquisition unit that acquires first sensing information indicating the position of a moving object in a work area in chronological order and a plurality of second sensing pieces of information indicating the ambient environmental conditions at different locations in the work area in chronological order; and a control unit that identifies the order of the series of individual tasks based on the first sensing information and identifies the work content of each of the series of individual tasks based on the second sensing information.

[0088] This allows for accurate identification of the work sequence.

[0089] (2) In the work sequence identification device of (1), the acquisition unit may acquire second sensing information from a plurality of proximity sensors located at different positions within the work area, and the control unit may select second sensing information to be used to identify the work content from among the plurality of second sensing information based on the position of the moving object indicated by the first sensing information and the positions of the plurality of proximity sensors.

[0090] This allows for accurate identification of the work content of individual tasks.

[0091] (3) In the work sequence identification device of (2), if the control unit has selected multiple second sensing information to be used to identify the work content based on the position of the moving object and the positions of multiple nearby sensors, it may decide to use one of the selected second sensing information to actually use.

[0092] This allows for accurate identification of the work content of individual tasks.

[0093] (4) In the work sequence identification device of (1), the control unit may divide the work area into multiple work locations and manage them, identify the work location where the moving object is located from among the multiple work locations based on the first sensing information, associate the work location with the work content, and generate work sequence information indicating the work sequence.

[0094] This work sequence information is useful for analyzing work processes and planning improvements such as equipment placement or workflow.

[0095] (5)(4) In the work sequence identification device, the control unit may detect the movement path of a moving object from a first predetermined position to a second predetermined position within the work area based on the first sensing information, and generate work sequence information by repeatedly identifying the work location and identifying the work content along the detected movement path.

[0096] By setting a first predetermined position and a second predetermined position according to the type of work, it is possible to generate work sequence information along the movement path according to the type of work.

[0097] (6) In the work sequence identification device of (5), the first sensing information and the second sensing information each include time information indicating the time of sensing, and the control unit may associate the first sensing information and the second sensing information based on the time information.

[0098] This allows for accurate identification of the sequence and content of individual tasks.

[0099] (7) In the work sequence identification device of (6), the control unit may measure the time required for each individual work based on the time information.

[0100] The measured time is useful for analyzing the work and planning improvements such as equipment placement or workflow.

[0101] (8) In the work sequence identification device of (7), the control unit may calculate at least one of the minimum value, average value, maximum value, standard deviation, and ranking of the time required for the same individual work.

[0102] The calculated minimum, average, maximum, standard deviation, and rankings are useful for analyzing the work and planning improvements such as equipment placement or workflow.

[0103] (9)(6) In the work sequence identification device, the control unit may calculate the operating rate of the moving object based on the proportion of the time within the total time of the movement path during which the amount of change in the position of the moving object is less than or equal to a first threshold and the amount of change in the surrounding environmental state is less than or equal to a second threshold.

[0104] Operating rates are useful for analyzing work processes and planning improvements such as equipment placement or workflow.

[0105] The work sequence identification device of (10)(1) may further include a display unit that displays the sequence of individual tasks, the work location, and the work content of a series of individual tasks, as well as at least one of the minimum, average, maximum, standard deviation, and ranking of the time required for each individual task.

[0106] This allows the user to visualize the work sequence.

[0107] (11) In the work sequence identification device of (1), the acquisition unit may acquire POS information indicating the purchased product, and the control unit may identify the work content based on the POS information and the second sensing information.

[0108] This allows for accurate identification of the work content of individual tasks.

[0109] (12) In the work sequence identification device of (1), the acquisition unit may acquire video data generated by a camera that photographs the work area as the first sensing information.

[0110] (13)(1) In the work sequence identification device, the ambient environmental conditions indicated by the second sensing information may include one or more of the following: sound, vibration, infrared radiation, electromagnetic waves, geomagnetic field, acceleration, temperature, humidity, illuminance, atmospheric pressure, and carbon dioxide concentration.

[0111] (14)(1) In the work sequence identification device, the work area is an area within the kitchen, and the series of individual tasks may include tasks related to cooking.

[0112] (15) The work sequence identification system of the present disclosure includes: a camera that photographs a work area and generates first sensing information that shows the position of a moving object within the work area in a time series; a plurality of proximity sensors that are placed at different locations within the work area and generate second sensing information that shows the state of the surrounding environment of the locations in a time series; and a work sequence identification device according to any one of (1) to (14) that identifies a work sequence including a series of individual tasks performed by a moving object based on the first sensing information and the plurality of second sensing information.

[0113] (16) A work sequence identification method of the present disclosure is a work sequence identification method that uses a calculation unit to identify a work sequence including a series of individual tasks, and includes the steps of: acquiring first sensing information that shows the position of a moving object in a work area in a time series; and a plurality of second sensing information that shows the ambient environmental conditions at different locations in the work area in a time series; and identifying the order of the series of individual tasks based on the first sensing information and identifying the work content of each of the series of individual tasks based on the second sensing information.

[0114] The work series identification method of (17)(16) may further include a display step that displays the order, location, and content of a series of individual tasks, and at least one of the minimum, average, maximum, standard deviation, and ranking of the time taken for each individual task.

[0115] The work series identification method of (18)(17) may include a display step of selecting one of the individual tasks from a series of individual tasks, a step of selecting one of the minimum, average, maximum, standard deviation, and ranking of the time taken for the selected individual task as a representative value, and a step of displaying the video of the individual task corresponding to the selected representative value.

[0116] (19) The program of this disclosure causes a computer to execute the work sequence identification method of (16).

[0117] The work sequence identification devices, work sequence identification systems, and work sequence identification device methods described in all claims of this disclosure are implemented by the cooperation of hardware resources, such as a processor, memory, and programs. [Industrial applicability]

[0118] The work sequence identification device and work sequence identification system of this disclosure are useful as devices for automatically identifying a work sequence that includes a series of individual tasks. [Explanation of symbols]

[0119] 1 Work sequence identification device 2 cameras 3. Nearby Sensors 4 POS terminals 11 Communications Department 12 Control Unit 13 Storage section 14 Input section 15 Display 16 bus 100 Work Sequence Identification System 120 Individual work identification devices

Claims

1. An analytical device that analyzes the behavior of people who purchase products, An acquisition unit that acquires first sensing information showing the location of a person purchasing goods within the store area in chronological order, second sensing information showing the surrounding environmental conditions in chronological order, and POS information. A control unit detects the movement information of the person from a first predetermined position to a second predetermined position within the store area based on the first sensing information, and identifies the product information to be purchased based on the POS information. Equipped with, The control unit identifies second sensing information within a predetermined range from a plurality of second sensing information based on the movement path information of the person indicated by the first sensing information, and analyzes the person's actions in the movement path information based on the detected movement path information, the probability of the content of the action determined based on the identified second sensing information, and the coefficient of the product indicated by the identified product information. Analyzer.

2. Based on the analyzed sequence of actions, we propose improvements. The analytical apparatus according to claim 1.

3. The control unit measures the time during which the amount of movement represented by the movement path information is less than or equal to a threshold. The analytical apparatus according to claim 1.

4. The aforementioned store area is divided into multiple locations for management, and the location is identified based on the traffic flow information and the division information. Based on the aforementioned location and POS information, a sequence of a person's actions is generated. The analytical apparatus according to claim 1.

5. The analyzed behaviors are input into the learning model as training data. The analytical apparatus according to claim 1.

6. The sensing information and the POS information each include time information. The control unit associates the sensing information and the POS information based on the time information. The analytical apparatus according to claim 1.

7. The control unit measures the time required for each action based on the time information. The analytical apparatus according to claim 6.

8. The control unit calculates at least one of the following: the minimum value, average value, maximum value, standard deviation, and ranking of the time taken for each action. The analytical apparatus according to claim 7.

9. The system further includes a display unit that shows the analyzed sequence of actions, the location of the person, the products purchased, and at least one of the minimum, average, and maximum values ​​of the time taken for each action. The analytical apparatus according to claim 1.

10. The acquisition unit acquires video data generated by a camera that photographs the store area as sensing information. The analytical apparatus according to claim 1.

11. The control unit, For each of the aforementioned individuals, the movement information is detected, Identifying a series of actions of the person in the aforementioned movement information, The aforementioned series of actions are associated with product information. The analytical apparatus according to claim 1.

12. The first sensing information is acquired by a remote sensor. The analytical apparatus according to claim 1.

13. A program that causes a computer to perform an analysis method to analyze the behavior of people who purchase products, The aforementioned analysis method The computer includes the steps of acquiring first sensing information that shows the location of a person purchasing goods within the store area in chronological order, second sensing information that shows the surrounding environment information in chronological order, and POS information. The computer includes the steps of detecting the movement information of the person from a first predetermined position to a second predetermined position within the store area based on the first sensing information, and identifying product information to be purchased based on the POS information, The computer includes the steps of: identifying a second sensing information within a predetermined range from a plurality of second sensing information based on the movement path information of the person indicated by the first sensing information; analyzing the person's behavior in the movement path information based on the detected movement path information, the probability of the content of the action determined based on the identified second sensing information, and the coefficient of the product indicated by the identified product information; A program that includes this.

Images