Information processing system

Abstract

To appropriately determine whether the cleaning work performed by the worker is being carried out using cleaning materials. [Solution] The information processing system 1000 includes an imaging unit 30 that photographs a predetermined monitoring area, a detection unit 21 that detects people and cleaning materials from the images captured by the imaging unit 30, and a determination unit 112 that determines the order of cleaning work for a person and the degree of cleaning based on a plurality of body areas set according to the size of the person detected by the detection unit 21 and the position of the cleaning material 80 within the body area.

Description

Technical Field

[0001] The present invention relates to an information processing system. In particular, the present invention relates to an information processing system for evaluating the cleaning work of an operator himself / herself when entering a clean room.

Background Art

[0002] Conventionally, at the entrance of a clean room with a high degree of cleanliness, such as a food factory or a manufacturing factory for precision instruments, a cleaning operation using cleaning tools is performed to remove foreign substances attached to the operator himself / herself or work clothes such as dust-proof clothing. The information processing apparatus disclosed in Patent Document 1 captures an operator with an imaging unit and extracts the joint positions of the operator based on the captured imaging image. Further, this information processing system detects a sanitary tool carried by the operator and recognizes the execution of a sanitary operation using the sanitary tool by the operator. Then, the information processing system identifies the body part during the sanitary operation based on the joint position and the position of the sanitary tool when the execution of the sanitary operation is recognized. Further, when the sanitary operation time for each body part exceeds the work completion time set for each body part, the completion of the work for each body part is determined.

[0003] Also, in the cleanliness management method for clean room entrants disclosed in Patent Document 2, the person area of the subject and the area of the roller held by the subject are extracted by background difference by comparing the acquired image including the subject captured in the inspection room with the background image. Further, by obtaining and analyzing the relative movement of the roller with respect to each body part in the person area by optical flow, it is determined whether the roller application has been appropriately performed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

[0005] However, in the technology disclosed in Patent Document 1, the completion of the work is determined when the working time for the body part being cleaned exceeds a set work completion time (judgment criterion). Furthermore, this work completion time is uniform regardless of the worker, and the worker's physique is not taken into consideration. Therefore, for example, if a worker has a large physique, the size of the body part being cleaned will be larger, and even if they clean for the same amount of time as a worker with a smaller physique, the cleaning may be insufficient. The same applies to the technology disclosed in Patent Document 2, where a rubbing judgment is made for each divided area, and the suitability of the completion of rubbing is determined for each part according to the number of rubbings. In this case as well, for example, if a worker has a large physique, the size of the body part being cleaned will be larger, and even if they clean for the same amount of time as a worker with a smaller physique, the cleaning may be insufficient.

[0006] This invention was made to solve the above-mentioned problems, and aims to provide an information processing system that can appropriately determine whether a worker's cleaning work is being performed properly, taking into account the worker's physical size. [Means for solving the problem]

[0007] The above objectives of the present invention are achieved by the following means.

[0008] (1) An imaging unit that photographs a designated monitoring area, A detection unit detects people and cleaning materials from the image captured by the aforementioned imaging unit, A determination unit determines the order of cleaning work on the person and the degree of cleaning based on a plurality of body areas set according to the size of the person detected by the detection unit and the position of the cleaning member within the body area. An information processing system equipped with the following features.

[0009] (2) The detection unit acquires the skeletal information of the person, The information processing system according to (1) above, wherein the determination unit sets a plurality of body areas corresponding to body parts, each having a size corresponding to the size of the person, based on the skeletal information.

[0010] (3) The body area is defined as having a standard size with a predetermined size for each body part. Based on the ratio of the length between predetermined skeletal information positions to a reference length, the reference size is changed, the body area is set with the changed size, and The information processing system according to (2) above, wherein the modified body area is positioned according to the predetermined position of the skeletal information.

[0011] (4) The information processing system described in (3) above, comprising a first reception unit that accepts changes by the user to the standard size for each body part and the position of the skeletal information.

[0012] (5) The information processing system according to (1) above, wherein the determination unit determines the degree of cleaning based on the time and trajectory of the cleaning member moving within the body area.

[0013] (6) The information processing system according to (1) above, further comprising a notification unit for notifying the determination result of the determination unit.

[0014] (7) The information processing system according to (2) above, wherein in a plurality of body areas, adjacent body areas are set to overlap each other within a certain range.

[0015] (8) The information processing system according to (7) above, wherein the determination unit determines that cleaning is in progress in any of the body areas when the cleaning member is located within that body area for a predetermined period of time or longer.

[0016] (9) Having a state transition model that shows the correct cleaning sequence for multiple body areas, The determination unit determines the order of the cleaning operation based on the position of the cleaning member within the body area using the state transition model, for the information processing system according to (7) above.

[0017] (10) In the case where the position of the cleaning member continuously exists within any of the body areas for a predetermined time or longer, the determination unit determines that the state has transitioned during the cleaning of the body area in the state transition model, for the information processing system according to (9) above.

[0018] (11) In the state transition model, when the state transitions in a direction set as an error, the determination unit detects an order error and counts it as the number of errors, for the information processing system according to (10) above.

[0019] (12) When the cleaning member exists within the overlapping range of the body areas, the determination unit does not determine the transition of the state, for the information processing system according to (11) above.

[0020] (13) The state transition model includes a second reception unit that receives a setting change of the correctness determination for the transition direction between states by the user, for the information processing system according to (11) above.

[0021] (14) The detection unit acquires the skeleton information of the person, When the determination unit determines that the action of the person based on the skeleton information is a gesture registered in advance, the determination unit determines the start and end of the cleaning operation, for the information processing system according to (1) above.

[0022] (15) A storage unit that stores time-series data associating the position information of the cleaning member from the start to the end of the cleaning operation, the classification result of the body area determined by the position information, and a time stamp, The determination unit includes an analysis unit that calculates a feature amount for determining the cleaning degree for each body area using the stored time-series data, for the information processing system according to (3) above.

[0023] (16) The feature amount is divided by the area of the body area set with the adjusted size, The determination unit determines the cleaning degree based on the divided feature amount, and the information processing system according to (15) above.

[0024] (17) The divided feature amount is normalized by a reference value set based on history data, and the information processing system according to (16) above.

[0025] (18) It includes a notification unit, The determination unit compares the normalized feature amount with a determination threshold for completion of cleaning to determine whether cleaning for each body area is completed or not completed, The notification unit outputs a display image indicating the result of the determination in a manner that can be identified by adding color separation or symbols to a pictogram indicating a person, and the information processing system according to (17) above.

[0026] (19) It includes a third reception unit that receives a change in setting of the determination threshold by a user, and the information processing system according to (18) above.

[0027] (20) The notification unit outputs a display image in which an image indicating the determination result of the cleaning degree for each body area is superimposed on the person in the video in a manner of color separation or indicating light and shade, and the information processing system according to (18) above.

[0028] (21) After the cleaning operation ends, the determination unit scores and records based on the normalized feature amount, After the cleaning operation ends, the notification unit feeds back the score to the person, and the information processing system according to (18) above.

[0029] (22) It includes a notification unit, The information processing system according to (2) above, wherein the notification unit outputs a display screen that includes a screen in which the skeletal information is superimposed on the person in the video, a screen showing the operating status of the information processing system, and a screen showing the judgment result. [Effects of the Invention]

[0030] The information processing system according to the present invention comprises an imaging unit that photographs a predetermined monitoring area, a detection unit that detects a person and cleaning materials from the image captured by the imaging unit, and a determination unit that determines the order of cleaning work for the person and the degree of cleaning based on a plurality of body areas set according to the size of the person detected by the detection unit and the position of the cleaning materials within the body areas. This makes it possible to appropriately determine whether the cleaning work by the worker is being performed appropriately, taking into account the size of the worker's body. [Brief explanation of the drawing]

[0031] The advantages and features provided by one or more embodiments of the present invention will be better understood from the following detailed description and accompanying drawings. However, these are for illustrative purposes only and are not intended to limit the present invention. [Figure 1] This is an overall block diagram showing the configuration of the information processing system according to this embodiment. [Figure 2] This is a schematic diagram showing workers in a monitoring area. [Figure 3] This is a functional block diagram showing the functions of the control unit. [Figure 4] Figure 4(a) is a schematic diagram showing an example of video data obtained by filming. Figure 4(b) is a schematic diagram showing skeletal information detected from the video data. [Figure 5A] This is a schematic diagram showing the defined body area. [Figure 5B] This is a schematic diagram showing the defined body area. [Figure 6A] This diagram shows the movements involved in cleaning the upper body. [Figure 6B] This diagram shows the movements involved in cleaning the lower body. [Figure 7] This is a schematic diagram illustrating the specific treatment 1 for cleaning components. [Figure 8] This is a schematic diagram illustrating the specific treatment 2 for cleaning components. [Figure 9] This is a schematic diagram showing the relationship between the position of the cleaning component and the specified cleaning area. [Figure 10] This is an example of time-series data. [Figure 11] This flowchart shows the cleaning sequence and cleaning level determination process performed by the information processing system. [Figure 12] This is a subroutine flowchart showing the process of step S08 in Figure 11. [Figure 13] This figure shows the state transition model used to determine the cleaning order. [Figure 14A] This is a subroutine flowchart showing the process in step S20 of Figure 12. [Figure 14B] This figure shows time-series data where label transitions are recorded. [Figure 14C] This data shows the range of each label in the time series data from Figure 14B. [Figure 15] This is an example of the evaluation results for cleaning order. [Figure 16] This is a subroutine flowchart showing the process of step S09 in Figure 11. [Figure 17] This is an example of an evaluation result for the level of cleanliness. [Figure 18A] This is a schematic diagram showing an example of a displayed image. [Figure 18B] This is a schematic diagram showing an example of a displayed image. [Figure 18C] This is a schematic diagram showing an example of a displayed image. [Figure 18D] This is a schematic diagram showing an example of a displayed image. [Modes for carrying out the invention]

[0032] Embodiments of the present invention will be described below with reference to the attached drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from the actual ratios.

[0033] Figure 1 is an overall block diagram showing the configuration of the information processing system 1000 according to this embodiment. Figure 2 is a schematic diagram showing a cleaner in a monitoring area. The information processing system 1000 includes a first information processing device 10, a second information processing device 20, and an imaging unit 30. These devices communicate with each other. The first information processing device 10 and the second information processing device 20 are PCs (personal computers) or servers (on-premise servers or cloud servers). For example, the first information processing device 10 is a PC or an edge server, and the second information processing device 20 is a cloud server. The imaging unit 30 is a camera or a device having a camera. The camera is composed of an image sensor and optical elements.

[0034] As shown in Figure 2, the imaging unit 30 is fixed to a wall or the like to photograph a predetermined monitoring area. The optical axis of the imaging unit 30 may be installed horizontally or slightly tilted downwards (within the range of 1 to 45 degrees). The imaging unit 30 starts and stops shooting under the control of the first information processing device 10. The imaging unit 30 outputs color or black and white video when it shoots. The imaging unit 30 shoots video (moving image) consisting of multiple time-sequentially continuous images (frames) at a frame rate of 10 to 60 fps (time-series video data). The video data obtained from shooting is sent to the first information processing device 10. For example, the imaging unit 30 outputs time-series video data at 30 fps.

[0035] The monitoring area is, for example, the anteroom of a food manufacturing line, a precision equipment manufacturing line, etc. (hereinafter simply referred to as the manufacturing line). Markers are placed on the floor of the monitoring area, and cleaning workers 90 (hereinafter simply referred to as the worker) are instructed to stand at the marker positions when cleaning their own bodies. The worker 90 is instructed to stand in front of the camera of the imaging unit 30 or slightly to the side of it within the monitoring area. In addition, when cleaning the back of the worker 90, they are instructed to position themselves so that the cleaning position is on the camera side. A fixed camera is used, and since the position where the subject (worker) stands is the same, the angle of view and shooting distance (distance to the subject) are always constant.

[0036] The cleaning operation is the process in which a worker cleans their own body using cleaning tools. More specifically, the worker uses a cleaning tool 80, such as a rotating roller with adhesive tape attached to a handle, to remove foreign matter such as fibers, dust, and hair adhering to the body surface by moving the cleaning tool 80 back and forth over their work clothes. The worker 90 follows the manual, and the cleaning sequence is from head to toe, working sequentially from top to bottom across the entire body. The state transition model (see Figure 13 below) is also described in accordance with the cleaning sequence described in this manual.

[0037] (First information processing device 10) Referring to Figure 1, the first information processing device 10 includes a control unit 11, an input unit 12, a display unit 13, a notification unit 14, a storage unit 15, and a communication unit 16. Although these components are electrically connected to each other, the connection relationships are omitted in Figure 1.

[0038] The control unit 11 consists of a CPU, GPU, RAM, ROM, etc. The input unit 12 is a keyboard, mouse, or touchscreen, and is used to input user instructions and various setting changes. The display unit 13 consists of a display such as an LCD and displays a display screen (GUI: Graphical User Interface) used for various inputs. The notification unit 14 is a speaker or buzzer. The display unit 13 and notification unit 14 may also be installed near the monitoring area in a visible location so that the cleaning status can be fed back to the worker 90 during cleaning work.

[0039] The memory unit 15 is composed of an SSD, HDD, etc. The memory unit 15 stores body area reference size, time-series data, history data, state transition model, trigger conditions, etc. These will be described later. The communication unit 16 is an interface circuit (e.g., a LAN card) for communicating with other devices such as the second information processing device 20 via a network.

[0040] (Second information processing device 20) The second information processing device 20 includes a control unit 21, a storage unit 22, and a communication unit 23. These components are electrically connected to each other, but the connection relationships are omitted in Figure 1. The hardware configurations of the control unit 21, storage unit 22, and communication unit 23 correspond to the control unit 11, storage unit 15, and communication unit 16 described above, respectively, so their explanation is omitted.

[0041] (Control unit 11, Control unit 21) Figure 3 is a functional block diagram showing the functions of the control units 11 and 21. The control unit 11 functions as an image acquisition unit 111 and a notification output unit 113 in cooperation with the communication unit 16. The control unit 11 also functions as a setting reception unit 114 in cooperation with the input unit 12 and the display unit 13. Furthermore, the control unit 11 functions as a determination unit 112 in cooperation with the control unit 21 of the second information processing device 20. The determination unit 112 functions as a body area setting unit 51, a cleaning area identification unit 52, a size correction unit 53, and a cleaning analysis unit 54 as sub-functions. The control unit 21 functions as a skeleton detection unit 211 and a cleaning material detection unit 212.

[0042] (Image acquisition unit 111) The image acquisition unit 111 acquires video data from the imaging unit 30. The video data is sent to the control unit 21 and the body area setting unit 51 of the determination unit 112. In this case, the image acquisition unit 111 may perform preprocessing on the video data before sending it to each functional component. Examples of preprocessing include convolution integral (smoothing, sharpening) and gradation correction using a one-dimensional lookup table.

[0043] (Judgment unit 112) The determination unit 112 determines the order of cleaning tasks performed by a person and the degree of cleaning based on multiple body areas and the position of the cleaning material within the body area. Details of the functions of the determination unit 112 will be described later.

[0044] (Notification output unit 113) The notification output unit 113 notifies the user, such as the worker or the manager who manages the worker, of the determination result made by the determination unit 112 via the display unit 13 or the notification unit 14.

[0045] (Settings reception unit 114) The setting reception unit 114 accepts user-submitted changes to settings such as the standard size of the body area, the placement position of the body area, the state transition model, trigger conditions, and the threshold for determining when cleaning is complete.

[0046] (Skeleton detection unit 211) The skeleton detection unit 211 analyzes the image sent from the image acquisition unit 111 to detect (estimate) the skeleton information of a person contained in the video data. The skeleton information may include positional information of multiple joint points, such as the eyes, nose, neck, shoulders, elbows, wrists, hips, knees, and ankles, which are the skeletal feature points (key joint points) of the person, and line segments connecting the joint points. Figure 4(a) is a schematic diagram showing an example of video data obtained by shooting. Figure 4(b) is a schematic diagram showing the skeleton information detected by the skeleton detection unit 211 from the video data. The positional information of the joint points may be the coordinates (X coordinate, Y coordinate) of the joint points on the captured image. In this specification, for illustrative purposes, some joint points may be omitted from the drawings.

[0047] The skeleton detection unit 211 estimates skeletal information using a pre-trained model for detecting joint points of a person from a rectangle 95 containing a person (hereinafter also referred to as the "person rectangle"). The person rectangle 95 is a region in the image that contains the joint points and nodes of a person. Examples of such pre-trained models include OpenPose (https: / / arxiv.org / abs / 1812.08008) and DeepPose (https: / / arxiv.org / abs / 1312.4659). The pre-trained model is stored in the memory unit 22. The skeletal information includes multiple joint points of the person, as well as the confidence level (a score indicating the likelihood of the estimation) for each joint point. In this embodiment, it is desirable to use joint points with a confidence level above a predetermined threshold. This is because, due to the nature of machine learning, if the confidence level is low, there is a high possibility that the detection result will not be correct. The skeleton detection unit 211 outputs the skeletal information to the body area setting unit 51 and the size correction unit 53.

[0048] (Setting the position and size of the body area) Figures 5A and 5B are schematic diagrams showing body areas set according to the size and build of a person (worker 90). The body area setting unit 51 sets multiple body areas corresponding to body parts determined from specific skeletal information of the worker 90, and sets the position, size, and inclination of each body area. In the example shown in Figure 5A, three body areas a1 to a3 (also referred to as "Head," "Upper," and "Lower," respectively) are set. Each body area a1 to a3 is rectangular in shape.

[0049] (Body area a1: Head) (Placement position: XY coordinate position) With respect to body area a1, the horizontal (X direction) and vertical (Y direction) coordinates are set so that the center of body area a1 is positioned at the joint point of the "nose".

[0050] (size) As shown below, the size correction unit 53 calculates correction coefficients r11 to r32, which are correction information, from the skeletal information received from the skeletal detection unit 211, using position information of a predetermined combination of joint points or distance information between joint points, and passes this to the body area setting unit 51. The body area setting unit 51 uses this to adjust or change the size of the body area.

[0051] The width of the rectangular (bounding box) body area a1 is set by the following formula.

[0052] Rectangle width = Correction factor r11 × Reference rectangle width sw (Equation 1) Here, the correction coefficient r11 = (shoulder width w1 + eye width w2) / (standard shoulder width ws1 + standard eye width ws2). Shoulder width w1 uses the distance (x coordinate) between the joint points of the left and right shoulders. Eye width w2 uses the distance between the joint points of the right and left eyes. Standard shoulder width ws1 and standard eye width ws2 use the average value of multiple users or the standard size of a worker, based on the same joint point position information. In addition, the standard rectangle width sw is a preset value and is stored in the memory unit 15 as the standard body area size.

[0053] The height (vertical width) of the rectangular body area a1 is set by the following formula.

[0054] Rectangle height = Correction factor r12 × Reference rectangle height sh (Equation 2) Here, the correction coefficient r12 = difference dy / standard difference dsy. The differences dy and dsy are the difference between the average height (y-coordinate) of the joint points of the left and right shoulders and the average height of the joint points of the left and right eyes. The standard difference dsy uses the average value of multiple users or the value of a worker with a standard size, based on the position information of similar joint points. The standard rectangle height sh is a preset value and is stored in the memory unit 15 as the standard body area size.

[0055] The user may be able to set the reference rectangular width sw, reference rectangular height sh, and placement position relative to the position of skeletal information for each body area through the setting reception unit 114. In this case, the setting reception unit 114 functions as a first reception unit.

[0056] (Slope) The body area a1 is tilted according to the average values ​​of the shoulder and eye tilt (the difference in angle from the horizontal line).

[0057] (Body areas a2, a3) The same process is used to set the XY coordinate position, size, and tilt of the rectangles in body areas a2 and a3. More specifically, in body area a2 (Upper), the position, size, and tilt are set based on the positional information of the joint points of the left and right shoulders and the left and right hips. For example, in body area a2, the XY coordinate position is set by the centroids of the four joint points of the left and right shoulders and left and right hips. Correction coefficients r21 and r22 are calculated from the width and height (y-coordinate difference) of these joint points, and the size is set using these coefficients. The tilt is also set using these joint points. Similarly, in body area a3 (Lower), the XY coordinate position is set by the coordinates of the four joint points of the left and right hips and left and right ankles. Correction coefficients r31 and r32 are calculated from the coordinates of these four joint points, and the size is set using these coefficients. The tilt is also set using these joint points.

[0058] (Area s0 of the reference body area, and area s2 of the body area after size correction) Figure 5A shows body areas a1 to a3 set for a worker with a standard body size. That is, the shoulder width w1, eye width w2, etc. are the same as the standard width, and the correction coefficients r11 to r31 (hereinafter collectively referred to as correction coefficient r) are all 1.0. In this case, the areas of body areas a1 to a3 are areas s0_1, s0_2, and s0_3 (hereinafter collectively referred to as area s0). Figure 5B shows body areas a1 to a3 set for a worker with a larger body size than the standard size, for example, worker 90 with skeletal positional relationships all being 10% larger or longer and r=1.1. Compared to Figure 5A, in the example shown in Figure 5B, the size of each body area a1 to a3 is set to be 10% larger in both length and width, according to the worker's body size. In this case, the areas of body areas a1 to a3 are areas s2_1, s2_2, and s2_3, respectively (hereinafter collectively referred to as area s2). In this example with a correction coefficient r=1.1, area s2=1.1 2 ×s0

[0059] (overlapping area) Figure 6A shows an example of upper body cleaning. Figure 6B shows an example of lower body cleaning. In the upper body cleaning operation shown in Figure 6A, the worker uses a cleaning tool to roll over the area from the abdomen to the waist. In the lower body cleaning operation shown in Figure 6B, the worker uses a cleaning tool to roll over the area from the waist to the knees. Thus, there are common areas to roll over in adjacent body parts. In the examples shown in Figures 6A and 6B, the cleaning tool enters the area around the waist in both the upper and lower body cleaning operations. Based on this understanding, in this embodiment, in order to prevent misjudgment of cleaning position and cleaning order, overlaps are made between adjacent body areas a1 to a3. By doing so, for example, if the waist were set as part of the lower body area without overlap, in the upper body cleaning operation shown in Figure 6A, the entry of the cleaning tool would be judged as having moved from the upper body to the lower body, increasing the likelihood of an incorrect cleaning order being determined. To prevent this from happening, in this embodiment, body areas a1 to a3 overlap with adjacent body areas.

[0060] (Cleaning member detection unit 212) The cleaning material detection unit 212 detects the position of the cleaning material 80 from the video data by the following specific processes 1 and 2. Figure 7 is a schematic diagram illustrating specific process 1. Specific process 1 is a method used when there is a high difference in brightness or chromaticity between the worker's 90 work clothes and the cleaning material 80. For example, this is the case when the work clothes are white and the cleaning material 80 is a low-brightness color other than white, such as black adhesive tape. The cleaning material detection unit 212 detects the area of ​​the cleaning material 80 (especially the adhesive tape) from the acquired video data by image processing such as binarization and contour extraction. At this time, the shape and size of the cleaning material 80 may be registered, and the cleaning material detection unit 212 may detect the cleaning material by pattern matching. Then, the centroid of the detected area is calculated. The cleaning material detection unit 212 sends the detected centroid position as the position of the cleaning material 80 to the determination unit 112.

[0061] Figure 8 is a schematic diagram illustrating specific process 2. Specific process 2 is performed in combination with specific process 1. The cleaning member detection unit 212 uses the skeletal information detected by the skeletal detection unit 211 to estimate the position of the right fist from the right elbow-right wrist vector and the position of the left fist from the left elbow-left wrist vector.

[0062] Then, an ROI (Region of Interest) is set centered on (1) the position of the right fist, (2) the position of the left fist, and (3) the coordinate position of the cleaning component extracted in the previous frame.

[0063] Next, the cleaning member detection unit 212 explores the three ROIs in the order of (1), (2), and (3), binarizes them using the processing of specific processing 1, and calculates the centroid position.

[0064] The cleaning member detection unit 212 then estimates the nearest object to be the cleaning member 80 based on the position coordinates of the cleaning member 80 on the front frame.

[0065] (Determination of entry of cleaning equipment into the body area) The following explanation of how to determine if a cleaning component enters a body area will refer to Figures 9 and 10. Figure 9 is a schematic diagram showing the relationship between the position of the cleaning component and the cleaning area identified therefrom, and Figure 10 is an example of time-series data. The time-series data associates the position information of the cleaning component, the classification result of the body area, and a timestamp, as described below. The time-series data describes the cleaning time (stay time) and movement trajectory of the cleaning component within the body area. The cleaning area identification unit 52 (see Figure 3) determines whether the cleaning component 80 is located within a body area, based on the position information (center of gravity) of the cleaning component 80 detected by the cleaning component detection unit 212 and the regions of multiple body areas a1 to a3 set by the body area setting unit 51. Then, a label is assigned based on the determination result and recorded as time-series data (data frame).

[0066] As shown in Figures 9 and 10, if the cleaning member 80 is located within any of the body areas a1 to a3, it is assigned the label of the respective body area a1 to a3. For example, if the cleaning member 80 is in body area a1, the label "Head" is assigned to the time-series data. Furthermore, if the cleaning member 80 is located in an overlapping area, it is assigned labels for multiple body areas. For example, if the cleaning member 80 is located in the overlapping area of ​​body areas a1 and a2, it is assigned the label "Head & Upper". If the cleaning member is not located in any body area, it is assigned the label "NaN". The example shown in Figure 10 shows that a label is assigned to each frame of 30fps.

[0067] (Process for determining cleaning order and level of cleaning) Next, referring to Figures 11 to 16C, the cleaning order and cleaning level determination process performed by the information processing system 1000 will be explained. Figure 11 is a flowchart of the cleaning order and cleaning level determination process.

[0068] (Steps S01, S02) The image acquisition unit 111 acquires video data captured by the imaging unit 30 of the monitoring area.

[0069] If a cleaning start trigger is received (YES), the control unit 11 proceeds to step S03. The trigger conditions for starting and ending the cleaning work are stored in the storage unit 15. For example, specific gestures are stored in the storage unit 15 as trigger conditions for starting and ending the cleaning work, respectively. If the control unit 11 determines that a specific gesture has been detected by the skeleton detection unit 211, it determines that the trigger condition has been met and decides to start the cleaning work. For example, a gesture could be a specific posture such as holding the cleaning member 80 above the head, as shown in Figure 5A. The trigger conditions for the start / end gestures may also be changed by the user through the setting reception unit 114. For example, the trigger condition could be changed to an action of swinging the cleaning member 80 held in the left hand from side to side above the head. Furthermore, the trigger conditions are not limited to these, and the trigger condition for starting or ending the work may be when the worker stands still at a designated position within the monitoring area. Another example is that the trigger condition may be activated by pressing a button placed around the monitoring area, or by holding an IC chip attached to the hand or wrist over a non-contact proximity sensor placed in the vicinity. Furthermore, the worker ID may be identified by holding the IC chip over the sensor. In this case, the operation may be started a predetermined time after the button is pressed or the hand or IC chip is held over the proximity sensor, or it may be terminated retroactively a predetermined time. For example, cleaning work could start 5 seconds after the button is pressed, or end 3 seconds before the button is pressed (in this case, the cleaning judgment results for the retroactive 3 seconds would be discarded).

[0070] (Steps S03, S04) The skeleton detection unit 211 detects skeleton information from the video data. The cleaning member detection unit 212 detects the position information of the cleaning member from the video data. These processes are as described above.

[0071] (Step S05) The size correction unit 53 calculates a correction coefficient r using the position information of a predetermined combination of joint points from the skeletal information. Then, the body area setting unit 51 changes the size of body areas a1 to a3 from the standard body area size using the correction coefficient r. Finally, the body area setting unit 51 places each of the size-corrected body areas a1 to a3 at the specified position and inclination using the position information of the joint points. The process described here is as explained with reference to Figures 5A and 5B.

[0072] (Steps S06, S07) The cleaning area identification unit 52 determines whether the cleaning member 80 has entered at least one of the body areas a1 to a3, based on the position (center of gravity) of the cleaning member 80 identified by the cleaning member detection unit 212 and the area information (size, position, inclination) of the body areas a1 to a3 set in step S05. The determination result is recorded as a label attached to the time-series data (see Figure 10).

[0073] (Steps S08, S09) The cleaning analysis unit 54 uses the time-series data recorded through the above processes to perform a cleaning order determination process (step S08) and a cleaning degree determination process (step S09). First, the process of step 08 will be explained, and then the process of step S09 will be explained.

[0074] (S08: Cleaning order determination process) The cleaning order determination process will be explained below with reference to Figures 12 to 14C.

[0075] Figure 12 is a subroutine flowchart showing the processing of step S08. Figure 13 is an example of a state transition model. This state transition model is stored in the memory unit 15. This state transition model may also be made available to the user through the setting reception unit 114, allowing them to change the settings for correct / incorrect judgment regarding the transition direction between states. In this case, the setting reception unit 114 functions as a second reception unit.

[0076] (Step S20) First, a label transition determination process is performed. Figure 14A is a subroutine flowchart showing the process in step S20.

[0077] (Step S201) The cleaning analysis unit 54 acquires time-series data. Here, the data length n corresponds to the number of frames in the time-series data. For example, the data length n is 2300.

[0078] (Steps S202-S206) Steps S202 through S206 are a loop. The initial value is i=0 and the end value is n-1.

[0079] In step S203, if the label of the acquired data i is a duplicate label indicating an overlapping area between body areas, i.e., "Head&Upper" or "Upper&Lower", or if it is NaN (null), the following processing is skipped. Then, in step S206, the label for the next data i+1 is acquired, and the processing from step S202 is executed. In all other cases, i.e., if the label is one of "Head", "Upper", or "Lower", the processing proceeds to step S204.

[0080] In step S204, if the same label appears N or more times consecutively (YES), a state transition is detected, and the process proceeds to step S205. Here, the same label is any label other than a duplicate label or a NaN label, i.e., any one of the labels "Head", "Upper", and "Lower" appears N or more times consecutively. N is a pre-set integer value, which can be any value from 2 to the tens. For example, N=3. On the other hand, if the previous label (i-1) and the current label are different, or if the number of consecutive identical labels is less than N, the process is skipped.

[0081] In step S205, label transitions are recorded. Figure 14B shows the time-series data in which label transitions have been recorded. In Range 0-3, the same label "Head" appeared three times in a row (the area enclosed by the dashed rectangle), so the start of the transition to the label "Head" is recorded by going back to Range 0. Similarly, in Range 200-202, the same label "Upper" appeared three times in a row, so the start of the transition to the label "Upper" is recorded in Range 200. In the preceding Range 199, the end of the transition of the label whose transition start was recorded immediately before is recorded. For example, in the example in Figure 14B, the end of the transition of the label "Head," whose transition start was recorded immediately before (Range 0), is recorded in Range 199. Through similar processing, the label "Head," which marked the start of the transition, is recorded in Range 500, and the label "Upper," which marked the end of the transition, is recorded in the preceding Range 499.

[0082] Figure 14C shows the range of each label for each row of the time-series data (Figure 14B). For example, Ranges 0-199 and 500-602 are labeled "Head", Ranges 200-499, 603-999, and 2000-2299 are labeled "Upper", and Range 1000-1999 is labeled "Lower".

[0083] This completes the subroutine flowchart in Figure 14A, and we return to the process in Figure 12.

[0084] (Steps S31, S32) The cleaning analysis unit 54 refers to the state transition model stored in the storage unit 15 and determines the correctness of the cleaning order from the label range (see Figure 14C) attached to the time-series data obtained by the label transition determination process in step S20. In the example in Figure 13, the transition from label "Upper" to "Upper" or "Lower" is a valid state transition, but the transition to "Head" is invalid, meaning that the cleaning was performed in the wrong order. In the data example in Figure 14C, multiple errors in the cleaning order are determined, and the number of order errors is counted. Figure 15 is an example of the output of the number of order errors determined by the cleaning analysis unit 54.

[0085] (S09: Determination of cleaning level) Next, we will explain step S09 in Figure 11. Figure 16 is a subroutine flowchart showing the processing of step S09.

[0086] (Step S20) First, the label transition determination process is performed. This process is explained with reference to Figures 14A to 14C.

[0087] (Steps S41-S43) The cleaning analysis unit 54 calculates the following feature quantities 1 to 3 using the label range (see Figure 14C) assigned to the time-series data and the coordinates of the cleaning member 80 within that range.

[0088] (Feature 1) The cleaning analysis unit 54 calculates the cleaning area ca for each body area. This involves finding the smallest convex polygon (convex hull) that encompasses all the position coordinates of the cleaning member 80 in each body area, and then calculating the area of ​​this convex polygon. Graham Scan or the Gift Wrapping Algorithm can be used to calculate the convex hull. Furthermore, various methods can be used to calculate the area of ​​the convex polygon; for example, it can be calculated by dividing it into triangles, calculating the area of ​​each triangle, and then summing them up.

[0089] (Feature 2) The cleaning analysis unit 54 calculates the cleaning time ct by the cleaning member 80 for each body area. The cleaning analysis unit 54 calculates this by counting the number of data points (frames) for each body area using time-series data.

[0090] (Feature 3) The cleaning analysis unit 54 calculates the length of the movement trajectory cl for each body area. The cleaning analysis unit 54 calculates the distance (Euclidean distance) between consecutive coordinate points of each body area in the time series data and sums these values.

[0091] (Step S44) The cleaning analysis unit 54 corrects feature quantities 1 to 3 by the size of the worker's physique. Specifically, the cleaning analysis unit 54 divides by the area s2 of each of the corrected body areas a1 to a3. In the example above, the area s2 of the corrected body area is r 2 It is double.

[0092] (Step S45) The cleaning analysis unit 54 evaluates the corrected feature quantities 1 to 3. The cleaning analysis unit 54 can evaluate using the following methods. Method 1 involves comparing each feature quantity with a predetermined threshold (Type 1 judgment threshold) for each body area, and determining that the cleaning of the body area is complete if the threshold is exceeded. Method 2 involves performing normalization. Specifically, the corrected feature quantities are normalized using the maximum and minimum values ​​determined by the historical data, with the maximum value set to 1 and the minimum value to 0. The completion of cleaning of the body area is then determined by comparing these normalized values ​​with a predetermined threshold (Type 2 judgment threshold). Here, the historical data is a dataset obtained by multiple workers on a manufacturing line where the information processing system 1000 is used. Figure 17 shows an example of the evaluation result of the degree of cleaning output by the cleaning analysis unit 54. Note that the Type 1 and Type 2 judgment thresholds may be changed by the user through the setting reception unit 114. In this case, the setting reception unit 114 functions as a third reception unit. This concludes the subroutine flowchart in Figure 16 regarding the determination of the cleaning level, and we return to the process in Figure 11.

[0093] (Step S10) The cleaning analysis unit 54 scores based on the number of errors in the cleaning sequence and the normalized value of the cleaning level. The cleaning analysis unit 54 also records the judgment result and time-series data linked to the worker's ID. Scoring may be done by simply summing the normalized scores, or by weighting the scores according to the importance of the body area. In this case, points may be deducted according to the number of errors in the cleaning sequence.

[0094] (Step S11) The notification output unit 113 performs notification output processing. Figure 18A is a first example of the display screen 131 displayed on the display unit 13 by the notification output processing. On the display screen 131, the image captured by the imaging unit 30 is displayed in real time in the display area d11, and the pictogram representing a person is displayed in a manner that allows the degree of cleaning to be visually indicated in the display area d12. Feature quantity 2 (cleaning time ct after correction (step S44)) is displayed in the display areas d12 and d13. The number of errors in the cleaning order (see Figure 15) and the total work time or elapsed time from the start of work are also displayed in the display area d13. In the display area d12, on the pictogram, the higher the degree of cleaning achievement (normalized value in the range of 0 to 1) for each body area, the darker the color or density, and the lower the degree of achievement, the lighter the color or density. Alternatively, instead of color or density, or together with it, text or symbols indicating the degree of achievement may be displayed. In the example shown in Figure 18A, the degree of cleaning is determined based only on feature 2 (cleaning time). However, the control unit 11 may also make judgments based on the other feature quantities 1 and 3 (area and trajectory).

[0095] Figure 18B shows another example of the display screen 132 displayed on the display unit 13 by the notification output processing of the notification output unit 113. The notification output unit 113 warns the worker in real time via the display screen 132 if an error in the cleaning sequence is detected. On the display screen 132, the fact that there was an error in the cleaning sequence is shown in display area d21, and the physical location of the error is shown in display area d22.

[0096] Figure 18B shows another example of the display screen 132 displayed on the display unit 13 by the notification output processing of the notification output unit 113. The notification output unit 113 warns the worker in real time via the display screen 132 if an error in the cleaning sequence is detected. On the display screen 132, the fact that there was an error in the cleaning sequence is indicated in display area d21, and the physical location where the error occurred is indicated in display area d22. The notification output unit 113 may also notify the worker of the cleaning sequence error by an audible warning from the notification unit 14 located on the side of the monitoring area. The warning sound may be an announcement informing the worker that an error has occurred or a buzzer sound.

[0097] Figure 18C shows another example of a display screen 133 displayed on the display unit 13 by the notification output processing of the notification output unit 113. In the display screen 133 of Figure 18C, the display area d31 shows the detected skeletal information and the judgment results of feature quantity 1 (area) superimposed on the real-time video. The superimposed judgment results are shown as convex polygons (convex hulls) color-coded according to the degree of achievement of feature quantity 1 for each body area. Display areas d32 and d33 are the same as the display screen 131 and will not be described. Display area d34 shows the status of the information processing system 1000. This display area d34 can display the status of the device, such as standby, cleaning, analysis, and analysis results. By displaying a screen like this display screen 133, which superimposes the skeleton and analysis results onto the video of a person, the worker can understand the basis of the analysis and easily grasp whether the cleaning is correct and how to improve it.

[0098] Figure 18D shows another example of the display screen 134 displayed on the display unit 13 by the notification output processing by the notification output unit 113. In Figure 18D, the scoring result determined in step S10 is displayed in display area d41. In addition, the worker information and the date and time (end time) of the work are displayed in display area d42, and the top ranking scores from the manufacturing line over a predetermined period are displayed as reference in display area d43.

[0099] This embodiment achieves the following effects through the following configuration.

[0100] (1) The information processing system according to this embodiment includes an imaging unit that photographs a predetermined monitoring area, and a detection unit that detects people and cleaning materials from the images captured by the imaging unit. It also includes a determination unit that determines the order of cleaning work for a person and the degree of cleaning based on a plurality of body areas set according to the size of the person detected by the detection unit and the position of the cleaning material within the body area. This makes it possible to appropriately determine the cleaning work performed by the worker on the worker themselves using the cleaning material.

[0101] (2) The determination unit also acquires the skeletal information of the person, and based on the skeletal information, sets up multiple body areas corresponding to body parts, each corresponding to the size of the person. This allows for appropriate determination of cleaning tasks, taking into account the size of the worker. For example, in the case of a large worker, the size of the body parts to be cleaned is larger than that of a small worker, so a wider area or a longer cleaning time is required compared to a small worker.

[0102] (3) In addition, in multiple body areas, adjacent body areas are set to overlap with each other within a certain range. The determination unit determines that a body area is being cleaned if the cleaning member is located within any of the body areas for a predetermined period of time or longer. The determination unit also does not determine a state transition if the cleaning member is located within the overlapping range of body areas. In this way, the cleaning order can be correctly determined. For example, in cleaning operations for the upper and lower body, if the cleaning member enters the area around the waist in both cases, misjudgment of the cleaning order can be appropriately prevented, and the cleaning order can be correctly determined.

[0103] The configuration of the information processing system 1000 described above is intended to illustrate the main features of the above embodiment, and is not limited to the above configuration; various modifications can be made within the scope of the claims. Furthermore, it does not preclude configurations that are typically found in image forming apparatuses.

[0104] The number of body areas and the cleaning sequence (state transition model) are set according to the manual for cleaning operations on the manufacturing line to which the information processing system is applied, and can be changed as appropriate. For example, the number of body areas is not limited to three, and may be divided into more. Also, in the example shown in Figure 1, the information processing system 1000 is shown as being composed of two information processing devices, a first and a second, but these devices may be configured as a single unit. Furthermore, in this embodiment, the imaging unit 30 is shown as being equipped with one camera that photographs the front side of the worker 90, but it is not limited to this, and a camera may also be installed on the opposite side of the monitoring area so that the rear side can be photographed simultaneously. In this case, the camera used for determination is switched depending on the position of the cleaning material.

[0105] Furthermore, in this embodiment, the determination unit 112 determined feature quantities 1 to 3, but it is possible to omit some of them instead of determining all of them. For example, feature quantity 3 may be omitted, and only feature quantities 1 and 2 may be used. Also, although an example was shown in which feature quantities 1 to 3 are divided by the area s2 of the corrected body area, feature quantities 2 and 3 (time, trajectory length) may be divided by the value of the area s2 raised to the power of 1 / 2.

[0106] Furthermore, the means and methods for performing various processing operations in the information processing system 1000 according to the above embodiment can be implemented by either a dedicated hardware circuit or a programmed computer. The program may be provided, for example, on a computer-readable recording medium such as a USB memory stick or a DVD (Digital Versatile Disc)-ROM, or it may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. The program may also be provided as a standalone application software, or it may be incorporated into the software of the device as a function of the device.

[0107] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are for illustrative purposes only and are not limiting. The scope of the present invention should be interpreted in accordance with the language of the appended claims. [Explanation of symbols]

[0108] 1000 Information Processing Systems 10. First Information Processing Device 11 Control Unit 111 Image acquisition unit 112 Judgment section 51 Body Area Setting Section 52 Cleaning part identification part 53 Size Correction Section 54 Cleaning Analysis Department 113 Notification output unit 114 Setting Reception Section 12 Input section 13 Display section 14 Notification Department 15 Storage section 16 Communications Department 20. Second Information Processing Device 21 Control Unit 211 Skeleton detection unit 212 Cleaning member detection unit 22 Memory section 23 Communications Department

Claims

1. An imaging unit that photographs a designated monitoring area, A detection unit detects people and cleaning materials from the image captured by the aforementioned imaging unit, A determination unit determines the order of cleaning work on the person and the degree of cleaning based on a plurality of body areas set according to the size of the person detected by the detection unit and the position of the cleaning member within the body area. An information processing system equipped with the following features.

2. The detection unit acquires the skeletal information of the person, The information processing system according to claim 1, wherein the determination unit sets a plurality of body areas corresponding to body parts, each having a size corresponding to the size of the person, based on the skeletal information.

3. The aforementioned body area has a standard size set for each body part, Based on the ratio of the length between predetermined skeletal information positions to a reference length, the reference size is changed, the body area is set with the changed size, and The information processing system according to claim 2, wherein the modified body area is arranged according to the predetermined position of the skeletal information.

4. The information processing system according to claim 3, further comprising a first receiving unit that accepts changes by the user to the standard size for each body part and the position of the skeletal information relative to the position.

5. The information processing system according to claim 1, wherein the determination unit determines the degree of cleaning based on the time and trajectory of the cleaning member's movement within the body area.

6. The information processing system according to claim 1, further comprising a notification unit for notifying the determination result of the determination unit.

7. The information processing system according to claim 2, wherein in a plurality of body areas, adjacent body areas are set to overlap each other within a certain range.

8. The information processing system according to claim 7, wherein the determination unit determines that cleaning is in progress in any of the body areas when the cleaning member is continuously located within any of the body areas for a predetermined period of time or longer.

9. It has a state transition model that shows the correct cleaning sequence for multiple body areas, The information processing system according to claim 7, wherein the determination unit determines the order of the cleaning work based on the position of the cleaning member within the body area using the state transition model.

10. The information processing system according to claim 9, wherein the determination unit determines, in the state transition model, that a state has transitioned during cleaning of a body area if the cleaning member is continuously located within any of the body areas for a predetermined period of time or longer.

11. The information processing system according to claim 10, wherein the determination unit detects a sequence error and counts it as the number of errors when the state transitions in the state transition model in a direction set as an error.

12. The information processing system according to claim 11, wherein the determination unit does not determine the state transition if the cleaning member is located within the overlapping range of the body areas.

13. The information processing system according to claim 11, further comprising a second receiving unit that accepts a user's change in the setting for determining correctness of the direction of transition between states in the state transition model.

14. The detection unit acquires the skeletal information of the person, The information processing system according to claim 1, wherein the determination unit determines, based on the skeletal information, that the person's movement is a pre-registered gesture, and determines the start and end of the cleaning work.

15. The system includes a storage unit that stores time-series data, which associates the position information of the cleaning member from the start to the end of the cleaning operation with the classification result of the body area determined by the position information and a timestamp. The information processing system according to claim 3, wherein the determination unit includes an analysis unit that calculates feature quantities for determining the degree of cleaning for each body area using the stored time-series data.

16. The aforementioned feature quantity is divided by the area of ​​the body area set to the adjusted size. The information processing system according to claim 15, wherein the determination unit determines the degree of cleaning based on the feature quantity after division.

17. The information processing system according to claim 16, wherein the feature quantity after division is normalized by a reference value set based on historical data.

18. Equipped with a news department, The determination unit determines whether cleaning is complete or incomplete for each body area by comparing the normalized feature quantity with a determination threshold for cleaning completion. The information processing system according to claim 17, wherein the notification unit outputs a display image showing the result of the determination in a manner that allows identification by assigning color coding or symbols to a pictogram representing a person.

19. The information processing system according to claim 18, further comprising a third reception unit that accepts changes to the setting of the judgment threshold by a user.

20. The information processing system according to claim 18, wherein the notification unit outputs a display image in which an image showing the determination result of the degree of cleaning for each body area is superimposed on the person in the video, either in different colors or in varying shades.

21. The determination unit scores and records the normalized feature quantities after the cleaning work is completed. The information processing system according to claim 18, wherein the notification unit provides feedback to the person with a score after the cleaning work is completed.

22. Equipped with a news department, The information processing system according to claim 2, wherein the notification unit outputs a display screen that includes a screen in which the skeletal information is superimposed on the person in the video, a screen showing the operating status of the information processing system, and a screen showing the determination result.

Images