Image processing device and image processing method

Abstract

The present invention provides an image processing device (13) and an image processing method with which: an image including a person (7) is acquired from an imaging device (11) worn by the person (7) and / or attached to an article worn by the person (7); the posture of the person (7) is detected from the image; from the image, displacement of the imaging device (11) is calculated using a reference position serving as reference for calculating displacement; and the posture is corrected using the displacement.

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method for recognizing human actions. [Background technology]

[0002] A method is known in which a pair of three-dimensional image data and two-dimensional image data captured by a camera from the photographer's side is acquired, the position and orientation of the camera is estimated based on the three-dimensional image data, skeletal posture parameters representing the skeletal posture of the photographer are extracted based on the estimated camera position and orientation, body type parameters representing the photographer's body type that have been determined in advance, and the three-dimensional image data, posture features of the photographer are extracted based on the skeletal posture parameters, image features are extracted based on the two-dimensional image data, and the photographer's behavior is recognized based on the posture features and image features and pre-trained classifier parameters for identifying the photographer's behavior (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-99982 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology has a problem in that if the mounting position of the camera attached to a person's head or the like shifts from its initial position, the human posture cannot be correctly recognized using image data acquired by the camera.

[0005] An object of the present invention is to provide an image processing device and an image processing method that are capable of correctly recognizing a human posture even if the mounting position of the camera is shifted. [Means for solving the problem]

[0006] The present invention solves the above problem by obtaining an image including a human from an imaging device attached to at least one of the human or an item worn by the human, detecting the posture of the human from the image, calculating a positional shift of the imaging device from the image using a reference position as a basis for calculating the positional shift, and correcting the posture using the positional shift. Effect of the Invention

[0007] According to the present invention, even if the mounting position of the camera is shifted, the posture of the person can be correctly recognized. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing one embodiment of an image processing system including an image processing device according to the present invention. [Figure 2A] 2 is a plan view showing a wearable terminal equipped with the imaging device of FIG. 1, and an operator's left forearm and left hand. [Figure 2B] FIG. 2 is a perspective view showing the head of a human being (operator) and a helmet to which a camera serving as the imaging device of FIG. 1 is attached. [Figure 2C] 2 is a plan view showing a camera which is an imaging device in FIG. 1 and the right arm of an operator. [Figure 3A] 2B is a diagram showing an example of an image of an operator captured by the wearable terminal of FIG. 2A. [Figure 3B] 2C is a diagram showing an example of an image of an operator captured by the camera of FIG. 2B. FIG. [Figure 3C] FIG. 2D is a diagram showing an example of an image of an operator captured by the camera of FIG. 2C. [Figure 4A] 3B is a diagram showing an example of a method for detecting the posture of the operator shown in FIG. 3A. [Figure 4B] 3C is a diagram showing an example of a method for detecting the posture of the operator shown in FIG. 3B. [Figure 4C] 3D is a diagram showing an example of a method for detecting the posture of the operator shown in FIG. 3C. [Figure 5A]2B is a diagram showing another example of an image of an operator captured by the wearable terminal of FIG. 2A. [Figure 5B] FIG. 2D is a diagram showing another example of an image of the operator captured by the camera of FIG. 2C. [Figure 6A] FIG. 4 is a diagram showing another example of an image of the operator captured by the camera of FIG. 2B (part 1). [Figure 6B] FIG. 2C is a diagram showing another example of the image of the operator captured by the camera of FIG. 2B (part 2). [Figure 7] 4 is a flowchart showing an example of an information processing procedure in the image processing system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of an image processing apparatus and an image processing method according to the present invention will be described with reference to the drawings.

[0010] [Image processing system] FIG. 1 is a block diagram showing an image processing system 1 according to the present invention. The image processing system 1 is a device that detects the behavior of an operator (corresponding to a human being according to the present invention) performing a predetermined operation. The predetermined operation is not particularly limited, and examples thereof include the operation of an in-vehicle device by a vehicle occupant (including a driver; the same applies below), the operation of a tool by a worker engaged in assembly work in a factory, and the operation of a maintenance tool by a mechanic engaged in vehicle maintenance at a vehicle dealership (hereinafter also referred to as a "dealer"). The operator (hereinafter also simply referred to as an "operator") whose behavior is detected by the image processing system 1 is not particularly limited, and examples thereof include a vehicle occupant, a factory worker, and a dealer mechanic. Examples of the behavior detected by the image processing system 1 include the behavior of a vehicle occupant reaching for a switch of an in-vehicle device such as an air conditioner to operate the in-vehicle device, the behavior of a worker at an assembly factory taking out a torque wrench from a toolbox, and the behavior of a dealer mechanic touching a switch that raises and lowers a lift that lifts a vehicle. By detecting the above-mentioned behaviors using the image processing system 1 of this embodiment, it is possible to confirm whether functions associated with the operation of in-vehicle equipment are functioning properly, whether the assembly work of workers engaged in assembly work at a vehicle assembly plant is being performed in accordance with predetermined procedures, and whether the maintenance work of mechanics engaged in servicing vehicles at a vehicle dealership is being performed in accordance with a manual.

[0011] As shown in FIG. 1, the image processing system 1 includes an imaging device 11, a display device 12, and an image processing device 13. The devices constituting the image processing system 1 are connected to each other by known means such as wired or wireless LAN in a state in which data can be exchanged between them. The number of imaging devices 11 and display devices 12 is not particularly limited as long as they are at least one or more. In addition, the imaging device 11 and the display device 12 do not need to be installed together with the image processing device 13, and may be installed at a location away from the image processing device 13. For example, the imaging device 11 and the display device 12 may be installed near an assembly line in an assembly factory, and the image processing device 13 may be installed in a central control room away from the assembly line or in a server at a remote location away from the assembly factory.

[0012] The imaging device 11 is a device for acquiring image data of an object existing around the operator, and is, for example, a camera equipped with an imaging element such as a CCD, an ultrasonic camera, an infrared camera, or the like. The object includes not only the operator but also objects existing around the operator. For example, the object includes switches and touch panels around the vehicle occupant, parts assembled by the worker and tools used, and a vehicle maintained by a dealer's mechanic. The imaging device 11 is installed in a position where the operator's body posture can be detected, such as the dashboard, roof, and seats of the vehicle, the assembly line of the assembly plant, the vicinity of the workbench and tools used by the worker, and the dealer's lift.

[0013] The imaging device 11 of this embodiment is attached to at least one of the operator who performs the above-mentioned predetermined operation and an article worn by the operator. In this case, the imaging device 11 is attached so that at least a part of the operator's body can be photographed. When attaching the imaging device 11 to the operator, for example, the imaging device 11 is attached to the operator's head, upper arm, forearm, abdomen, thigh, lower leg, etc. Examples of attachment methods include a method of wrapping and fixing the imaging device 11 around the operator's body using a hook-and-loop fastener or a buckle, a method of attaching the imaging device 11 to the operator's body using an adhesive tape, and a method of attaching the imaging device 11 to the operator's body using an adhesive.

[0014] As an example, when the imaging device 11, the display device 12, and the image processing device 13 are integrated into one wearable terminal 3 such as a smart watch, the terminal is worn as shown in FIG. 2A. FIG. 2A is a plan view showing the operator's left forearm 73b and left hand 73c, and the left forearm 73b is covered with clothing. In this case, the wearable terminal 3 is worn by wrapping a band 31 having a hook-and-loop fastener around a part of the operator's left forearm 73b near the wrist. The wearable terminal 3 includes an imaging device 11, which captures an image within a field of view 4 indicated by a dashed line in FIG. 2A. This allows an image including the body of the operator wearing the wearable terminal 3 to be captured.

[0015] On the other hand, the article worn by the operator refers to the clothes and protective gear worn by the operator when performing a predetermined operation. Examples of the clothes and protective gear include jackets, coats, wristwatches, rings, earrings, and pierced earrings worn by vehicle occupants, and work clothes, hats, safety shoes, helmets, masks, and protective glasses worn by workers at assembly plants and mechanics at dealerships. Methods of attaching the imaging device 11 to the clothes and protective gear worn by the operator include, for example, a method of fixing the imaging device 11 to the coat or work clothes using a hook-and-loop fastener, a method of attaching a member such as a clamp to the hat, helmet, mask, protective glasses, etc. and gripping the imaging device 11 with the member, and a method of attaching the imaging device 11 to the wristwatch or safety shoes using an adhesive tape or adhesive.

[0016] As an example, when the imaging device 11 is a CCD camera equipped with a wide-angle lens, a fisheye lens, or an ultra-wide-angle lens, it is attached as shown in FIG. 2B. FIG. 2B is a perspective view showing the operator's head 71 and the helmet 5 attached to the head 71. In this case, the camera 6 is attached to the brim 51 of the helmet 5 worn by the operator, facing downward so that the operator's body can be photographed. The method of attaching the camera 6 is such that a bracket 52 for holding the camera 6 is attached to the helmet 5, and the bracket 52 holds the camera 6. Alternatively, a part for clamping the camera 6 may be provided on the helmet 5, and the part may be integrally molded as the helmet 5 to clamp the camera 6. The camera 6 captures an image within the range of the field of view 4a shown by the dashed line in FIG. 2B. This allows an image including the body of the operator wearing the helmet 5 to be captured.

[0017] In addition, when the camera 6 is attached to the operator, it is attached as shown in FIG. 2C, for example. FIG. 2C is a plan view showing the right arm of the operator, which is composed of the upper arm 74a of the right arm, the forearm 74b of the right arm, and the right hand 74c. In this case, the camera 6 is attached by wrapping a band 61 having a hook-and-loop fastener around a part of the forearm 74b of the right arm close to the elbow. The camera 6 captures an image in the range of the field of view 4b shown by the dashed line in FIG. 2C. This allows an image including the right hand 74c of the operator wearing the camera 6 to be captured. In FIG. 2C, the camera 6 is attached so that the back side of the right hand 74c of the operator can be captured, but the camera 6 may be attached so that the palm side of the right hand 74c of the operator can be captured. Which side is to be captured is appropriately selected depending on the action of the operator to be detected. In the example of FIG. 2A, the wearable terminal 3 is attached to the left arm, and in the example of FIG. 2C, the camera 6 is attached to the right arm, but the wearable terminal 3 and the camera 6 may be attached to either the left or right arm. Furthermore, the mounting positions of these devices can be set to any suitable position within a range where the actions of the operator can be appropriately detected.

[0018] Returning to FIG. 1, the display device 12 is a device for notifying the operator of the action detected by the image processing device 13. The display device 12 is, for example, a liquid crystal display, a projector, or the like, and may be equipped with a speaker. The display device 12 is installed in a position near the operator, such as on the dashboard of a vehicle or the work place of a worker in an assembly plant, where the display device 12 can notify the operator of necessary information, but if there is a supervisor who monitors the operation of the operator, the display device 12 is installed near the supervisor. In this case, if the supervisor is located away from the operator, the display device 12 is installed at a position away from the operator. The display device 12 may be attached to the operator as a wearable terminal such as glasses. Furthermore, instead of the display device 12, only a speaker that emits an alarm sound in response to the detected action may be used.

[0019] The image processing device 13 is a device, such as a computer, for estimating the behavior of an operator performing a predetermined operation from the posture of the operator. In particular, the image processing device 13 can take into consideration the positional deviation of the imaging device 11 when detecting the posture of the operator. The image processing device 13 cooperates with the imaging device 11 and the display device 12 to obtain image data from the imaging device 11 at predetermined time intervals and process the obtained image data to detect the posture of the operator, correct the positional deviation of the imaging device 11, and estimate the behavior of the operator.

[0020] In order to realize the data acquisition function, posture detection function, position deviation correction function, and behavior estimation function, the image processing device 13 includes a CPU (Central Processing Unit) 131 which is a processor, a ROM (Read Only Memory) 132 in which a program is stored, and a RAM (Random Access Memory) 133 which functions as an accessible storage device. The CPU 131 is an operating circuit for functioning as the image processing device 13 by executing the program stored in the ROM 132. Note that the image processing device 13 does not need to be provided together with the imaging device 11 and the display device 12, and may be provided in a server located remotely away from these devices.

[0021] [Processing section] The program used in the image processing device 13 includes a processing unit 2 which is a functional block for realizing the functions of the image processing device 13 described above. The processing unit 2 controls the imaging device 11 and the display device 12 to cooperate with each other, thereby realizing a function of acquiring image data including the operator from the imaging device 11 (data acquisition function), a function of processing the acquired image data, correcting the positional deviation of the imaging device 11, and then detecting the posture of the operator (posture detection function and positional deviation correction function), and a function of estimating the behavior of the operator from the posture of the operator (behavior estimation function). As functional blocks corresponding to these functions, the processing unit 2 includes an acquisition unit 21, a detection unit 22, a calculation unit 23, a correction unit 24, an estimation unit 25, and an output unit 26, as shown in FIG. 1. Each unit is extracted and shown in FIG. 1 for convenience.

[0022] 1 includes all of the above-mentioned functional blocks, but it is not necessary for a single image processing device 13 to include all of the functional blocks, and some of the above-mentioned functional blocks may be provided in other devices included in the image processing system 1, or in another information processing device (not shown). For example, in the image processing system 1 of FIG. 1, the detection unit 22 may be provided in the imaging device 11. In this case, the function of the detection unit 22 is executed using the CPU, ROM, and RAM of the imaging device 11.

[0023] In addition, it is not necessary to execute all the processes of each functional block in a single device, and the functions of each functional block may be realized across a plurality of devices connected in a state in which data can be exchanged. For example, in the image processing system 1 of Fig. 1, among the processes executed in the calculation unit 23, a part of the processes may be executed in the imaging device 11, and the remaining processes may be executed in the image processing device 13. In this case, a part of the processes for realizing the functions of the calculation unit 23 is performed using the CPU, ROM, and RAM of the imaging device 11.

[0024] The functions performed by each functional block of the processing unit 2 will be described below.

[0025] The acquisition unit 21 has a function of acquiring an image including the operator from the imaging device 11. As shown in Figs. 2A to 2C, the imaging device 11 of this embodiment is installed at a position and orientation where it can capture an image including the operator's body, so that the acquisition unit 21 can acquire an image including the operator by acquiring data of images captured by the imaging device 11 at a predetermined time interval. Moreover, the image data acquired by the acquisition unit 21 is not limited to still images, and may be time-series images such as a video. Examples of images acquired by the acquisition unit 21 are shown in Figs. 3A to 3C.

[0026] Fig. 3A is an image captured by the wearable device 3 shown in Fig. 2A, and includes an operator 7 who is a vehicle occupant. The operator 7 who is a vehicle occupant is wearing the wearable device 3 on his left forearm 73b. Therefore, when the operator 7 is photographed by the wearable device 3, the left upper arm 73a and the left forearm 73b are photographed on the right side, and the right upper arm 74a and the right forearm 74b are photographed on the left side, as shown in Fig. 3A. In addition, the entire head 71 and neck 72 of the operator 7 are photographed, and part of the torso 75 is also photographed.

[0027] FIG. 3B is an image captured by the camera 6 shown in FIG. 2B, and includes an operator 7 who is a worker at a vehicle assembly plant. The operator 7 who is a worker wears the camera 6 facing downward on the brim 51 of the helmet 5 worn on the head 71. Therefore, when the operator 7 is photographed by the camera 6, the part of the head 71 above the brim 51 (i.e., the part covered by the helmet 5) cannot be photographed. Therefore, as shown in FIG. 3B, an image cannot be acquired for the upper part of the head 71. A neck 72 and a torso 75 are photographed below the head 71, a right upper arm 74a, a right forearm 74b, and a right hand 74c are photographed on the left side of the torso 75, and a left upper arm 73a, a left forearm 73b, and a left hand 73c are photographed on the right side of the torso 75. The camera 6 photographs the operator 7 from the front direction of the operator 7, so the left and right are reversed as in FIG. 3A. Also, below the torso 75, a part of a left leg 76 and a part of a right leg 77 are photographed.

[0028] Fig. 3C is an image captured by the camera 6 shown in Fig. 2C, and includes a right hand 74c of an operator who is a worker at a vehicle assembly plant. The operator is wearing the camera 6 on a part of the right forearm 74b near the elbow, so when the right hand 74c is photographed by the camera 6, the wrist side of the right forearm 74b and the right hand 74c are captured as shown in Fig. 3C.

[0029] Returning to FIG. 1, the detection unit 22 has a function of detecting the posture of the operator 7 from the image acquired by the acquisition unit 21. The posture of the operator 7 refers to the body stance or posture of the operator 7, and specifically refers to the position of each joint of the body of the operator 7 and the relationship of the joints connecting to each other. For example, if the position of each joint of the body of the operator 7 and the body parts connecting the joints are clear, the body stance of the operator can be geometrically represented by plotting points at coordinates corresponding to the position of each joint in a three-dimensional space defined by an orthogonal coordinate system and connecting the joints (points) connected by the relevant parts with straight lines.

[0030] As an example, when the image shown in FIG. 3A is acquired by the acquisition unit 21, the detection unit 22 performs processing as shown in FIG. 4A to detect the posture of the operator 7. In FIG. 4A, first, the joints of the operator 7 are detected from the operator 7 included in the image by pattern matching or the like. Specifically, in the input image data, characteristic parts such as parts where the body orientation of the operator 7 is changing are extracted, and the extracted characteristic parts are compared with the class dictionary stored in the database 14 to determine which class the characteristic parts belong to. If a class for each joint is set in the class dictionary, it is possible to determine whether or not the characteristic parts correspond to joints.

[0031] Next, points are plotted on the parts determined to be joints. Specifically, as shown in Fig. 4A, a point P1 is plotted on a part corresponding to the joint of the neck 72 of the operator 7, a point P2 is plotted on a part corresponding to the joint of the right shoulder, a point P3 is plotted on a part corresponding to the joint of the right elbow, a point P4 is plotted on a part corresponding to the joint of the left shoulder, and a point P5 is plotted on a part corresponding to the joint of the left elbow. Next, a straight line corresponding to the right shoulder is drawn between points P1 and P2, a straight line corresponding to the upper arm 74a of the right arm is drawn between points P2 and P3, a straight line corresponding to the left shoulder is drawn between points P1 and P4, and a straight line corresponding to the upper arm 73a of the left arm is drawn between points P4 and P5.

[0032] In addition, the image shown in FIG. 3A does not include the right and left hands of the operator 7, but since there is no joint between the elbow joint and the wrist, it is estimated that the right hand is on the extension line of the right forearm 74b included in FIG. 4A, and the left hand is on the extension line of the left forearm 73b. Therefore, as shown in FIG. 4A, a straight line is drawn from point P3 toward the position of the estimated right hand (not shown), and a straight line is drawn from point P5 toward the position of the estimated left hand (not shown). These straight lines pass through the centers of the right forearm 74b and the left forearm 73b in a direction perpendicular to the length direction of the right forearm 74b and the left forearm 73b, for example. In addition, although the image shown in FIG. 3A does not include the part of the trunk 75 below the waist, a straight line is drawn from point P1 corresponding to the joint of the neck 72 through the center parts of the left and right of the trunk 75 to represent the position of the trunk 75. In this way, the detection unit 22 detects the posture of the operator 7 by representing the position of each joint in the body of the operator 7 included in FIG. 3A and the connections between the joints using points P1 to P5 shown in FIG. 4A and straight lines drawn from each point.

[0033] As another example, when the image shown in FIG. 3B is acquired by the acquisition unit 21, the detection unit 22 performs processing as shown in FIG. 4B to detect the posture of the operator 7. In FIG. 4B, first, joints are detected by processing similar to the pattern matching described in FIG. 4A. Next, points are plotted on the parts determined to be joints as in the case of FIG. 4A. Specifically, as shown in FIG. 4B, a point P6 is plotted on a part corresponding to the joint of the neck 72 of the operator 7, a point P7 is plotted on a part corresponding to the joint of the right shoulder, a point P8 is plotted on a part corresponding to the joint of the right elbow, a point P9 is plotted on a part corresponding to the joint of the right wrist, a point P10 is plotted on a part corresponding to the joint of the left shoulder, a point P11 is plotted on a part corresponding to the joint of the left elbow, and a point P12 is plotted on a part corresponding to the joint of the left wrist. Furthermore, point P13 is plotted at the part corresponding to the hip joint, point P14 is plotted at the part corresponding to the right hip joint, point P15 is plotted at the part corresponding to the right knee joint, point P16 is plotted at the part corresponding to the left hip joint, and point P17 is plotted at the part corresponding to the left knee joint.

[0034] Next, a line corresponding to the right shoulder is drawn between points P6 and P7, a line corresponding to the right upper arm 74a is drawn between points P7 and P8, a line corresponding to the right forearm 74b is drawn between points P8 and P9, a line corresponding to the left shoulder is drawn between points P7 and P10, a line corresponding to the left upper arm 73a is drawn between points P10 and P11, and a line corresponding to the left forearm 73b is drawn between points P11 and P12. Furthermore, a line corresponding to the torso 75 is drawn between points P6 and P13, a line corresponding to the groin is drawn between points P13 and P14 and between points P13 and P16, a line corresponding to the right leg 77 is drawn between points P14 and P15, and a line corresponding to the left leg 76 is drawn between points P16 and P17.

[0035] Further, the image shown in FIG. 3B does not include the right and left ankles of the operator 7, but since there is no joint between the knee joint and the ankle, it is estimated that the right ankle is on an extension of the right leg 77 included in FIG. 4B, and the left ankle is on an extension of the left leg 76. Therefore, as shown in FIG. 4B, straight lines are drawn from points P15 and P17 toward the estimated positions of both ankles (not shown). These straight lines pass through the centers of the right leg 77 and the left leg 76 in a direction perpendicular to the length direction of the right leg 77 and the left leg 76, for example. In this way, the detection unit 22 detects the posture of the operator 7 by expressing the positions of the joints of the body of the operator 7 included in FIG. 3B and the relationship of the connections between the joints by points P6 to P17 shown in FIG. 4B and straight lines drawn from each point.

[0036] As yet another example, when the image shown in FIG. 3C is acquired by the acquisition unit 21, the detection unit 22 performs processing as shown in FIG. 4C to detect the posture of the operator's right hand 74c. In FIG. 4C, first, joints are detected by processing similar to the pattern matching described in FIG. 4A. Next, points are plotted on the parts determined to be joints as in the case of FIG. 4A. Specifically, as shown in FIG. 4C, point P18 is plotted on the part corresponding to the joint of the operator's right wrist, points P19-P20 are plotted on the part corresponding to the joint of the thumb, points P21-P23 are plotted on the part corresponding to the joint of the index finger, points P24-P26 are plotted on the part corresponding to the joint of the middle finger, points P27-P29 are plotted on the part corresponding to the joint of the ring finger, and points P30-P32 are plotted on the part corresponding to the joint of the little finger. Next, draw straight lines corresponding to the palm between points P18 and P19, between points P18 and P21, between points P18 and P24, between points P18 and P27, and between points P18 and P30. Furthermore, draw straight lines corresponding to the thumb between points P19 and P20, draw straight lines corresponding to the index finger between points P21 and P22, and between points P22 and P23, draw straight lines corresponding to the middle finger between points P24 and P25, and between points P25 and P26, draw straight lines corresponding to the ring finger between points P27 and P28, and between points P28 and P29, and draw straight lines corresponding to the little finger between points P30 and P31, and between points P31 and P32.

[0037] Further, the image shown in FIG. 3C does not include the right elbow of the operator, but since there is no joint between the elbow and the wrist, it is estimated that the right elbow is present on an extension line of the right forearm 74b included in FIG. 4C. Therefore, as shown in FIG. 4C, a straight line is drawn from point P18 toward the position of the estimated right elbow (not shown). For example, this straight line passes through the center of the right forearm 74b in a direction perpendicular to the length direction of the right forearm 74b. In this way, the detection unit 22 detects the posture of the right hand 74c of the operator by expressing the positions of the joints of the right hand 74c of the operator included in FIG. 3C and the relationship of the connections between the joints by points P18 to P32 shown in FIG. 4C and the straight lines drawn from each point.

[0038] In addition to the above-mentioned methods, the detection unit 22 can detect the posture of the operator 7 based on the relative positional relationship between the body parts of the operator 7 and objects around the operator 7. For example, the detection unit 22 calculates where the head 71, arms, etc. of the operator 7 are located with respect to the ground on which the operator 7 is standing, and detects the posture of the operator 7 based on the calculation result. Alternatively or in addition to this, the detection unit 22 can detect the posture from the orientation of the body parts of the operator 7. For example, the posture of the operator 7 is detected from the position of the eyes (gaze), the orientation of the hands, the orientation of the feet, and the orientation of the torso 75 of the operator 7.

[0039] In addition, the joints at which points are set by the detection unit 22 are not limited to the joints shown in Figs. 4A to 4C. In Figs. 4A and 4B, joints finer than those shown in Figs. 4A and 4B may be detected and more points may be plotted. For example, as shown in Fig. 4C, points may be plotted at parts corresponding to the joints of the fingers of the left hand 73c and the right hand 74c to detect how the fingers of the left hand 73c and the right hand 74c are posed. This allows for the detection of fine postures, such as whether or not a vehicle occupant is touching an on-board device of the vehicle, or whether or not a worker on an assembly line is holding a tool.

[0040] When detecting the posture of the operator 7 from the image acquired by the acquisition unit 21, the detection unit 22 may use a posture estimation model that has been trained in advance to estimate the posture of the operator 7 from an image including the operator 7. The posture estimation model is stored, for example, in the database 14 shown in FIG. 1. In the posture estimation model, an image including the operator 7 and a posture of the operator 7 detected in the image are associated so as to correspond to each other, and when certain image data is input, a posture of the operator 7 corresponding to the input image data is output. The posture estimation model is machine-trained by the first machine learning unit 221 shown in FIG. 1. Specifically, the posture estimation model is trained to detect an appropriate posture for input image data and output a result using past images and detection results of the posture of the operator 7 stored in the database 14.

[0041] The posture estimation model may be a trained model that has undergone machine learning. The trained model is a model that has been trained in advance by machine learning so that appropriate output data can be obtained for certain input data, and includes at least a program that performs calculations from input data to obtaining output data, and weighting coefficients (parameters) used in the calculations. When image data acquired by the acquisition unit 21 is input as input data, the trained model causes a computer (particularly, the CPU 131, which is a processor) to function so that output data including the posture of the operator 7 is output based on the input data. By using such a trained model, it is possible to detect the posture of the operator 7 performing an operation other than the learned operation.

[0042] The trained model of the present embodiment is not particularly limited, but may be, for example, a neural network. The neural network includes an input layer, an intermediate layer, and an output layer, and each layer includes at least one neuron. The input layer receives input data including image data acquired by the acquisition unit 21, and outputs the input data to the intermediate layer. The intermediate layer extracts data of the operator 7 from the data input from the input layer. Next, the posture is detected from the extracted data of the operator 7. The output layer outputs the data input from the intermediate layer as output data including posture data. As parameters in the intermediate layer, the positions of the joints of the operator 7 and the connection relationship between the joints are considered.

[0043] The calculation unit 23 has a function of identifying a reference position as a reference for calculating the positional deviation from the image acquired by the acquisition unit 21, and calculating the positional deviation of the imaging device 11 using the identified reference position. The imaging device 11 of this embodiment is attached to at least one of the operator 7 and an article worn by the operator 7, but the attached imaging device 11 is not necessarily completely fixed, and the attachment position of the imaging device 11 may move from the position where it was originally attached while the operator 7 is performing a predetermined operation. For example, in FIG. 2A, when the band 31 rotates with respect to the wrist and the wearable terminal 3 rotates and moves toward the thumb side of the left hand 73c, the acquisition unit 21 acquires an image in which the operator 7 rotates counterclockwise (in the direction of the arrow 8) as shown in FIG. 5A. Also, in FIG. 2C, when the band 61 loosens and the camera 6 rotates clockwise, the acquisition unit 21 acquires an image in which the right hand 74c rotates clockwise (in the direction of the arrow 9) as shown in FIG. 5B. In these cases, there is a risk that the posture of the operator 7 may not be detected correctly by the detection unit 22, so the calculation unit 23 calculates the positional deviation of the mounting position of the imaging device 11, and the correction unit 24, which will be described later, corrects the posture detected by the detection unit 22. As a result, the posture of the operator 7 is detected correctly.

[0044] The positional deviation of the imaging device 11 is calculated with respect to a reference position in a predetermined state. The predetermined state is, for example, the initial state when the imaging device 11 is worn (hereinafter also referred to as "initial state"), the state when the operator 7 starts a predetermined operation, a state a predetermined time before a certain point in time (for example, the present), or a state at a certain time. The amount by which the reference position has moved (deviation) with respect to the reference position in these states is calculated, and the calculation result is regarded as the positional deviation of the imaging device 11. When calculating the positional deviation, the calculation unit 23 identifies a part of the body of the operator 7 included in the image as the reference position, and calculates the positional deviation using the identified body part.

[0045] The above-mentioned part of the body is a part of the body that is identified to calculate the positional deviation as simply as possible, and the part itself does not have a movable part such as a joint. Specifically, when the imaging device 11 is attached to the operator 7, the part is a part that does not have a joint between the part where the imaging device 11 is attached, and when the imaging device 11 is attached to an article worn by the operator 7, the part is a part that does not have a joint between the part where the imaging device 11 is in contact with the article worn by the operator 7. The reason why the part itself does not have a part that moves by a joint is that when calculating the positional deviation using the body part as the reference position, it is necessary to distinguish whether the part at the reference position has moved due to the movement of a joint or the part at the reference position has moved due to a positional deviation of the imaging device 11, which makes it difficult to accurately calculate the positional deviation. Also, the reason why there is no joint between the part where the imaging device 11 is attached and the part in contact with the article worn by the operator 7 is that if there is a joint between the imaging device 11 and the identified part, it becomes difficult to distinguish whether the movement of the part set as the reference position is due to the movement of a joint or due to a positional deviation of the wearing position.

[0046] However, since the position where the imaging device 11 is attached cannot be used to calculate the positional deviation of the part where the imaging device 11 is attached, the part of the body of the operator 7 where the imaging device 11 is attached and the part in contact with the article worn by the operator 7 are excluded from the candidates for the part that will be the reference position. Here, the part where the imaging device 11 is attached and the part in contact with the article worn by the operator 7 are part of the body of the operator 7. In other words, the part identified as the part that will be the reference position and the part where the imaging device 11 is attached may be the same part. Also, the part identified as the part that will be the reference position and the part in contact with the article worn by the operator 7 may be the same part.

[0047] When detecting a part to be a reference position, for example, a preset part of the body of the operator 7 included in the image data is detected by pattern matching. Then, when the preset part is identified, the part is set as a reference position for calculating the positional deviation. As an example, as shown in FIG. 2A, when the wearable terminal 3 is worn on the wrist of the left hand 73c, the forearm 73b of the left arm existing between the wrist and the left elbow joint is set in advance as a part to be a reference position. The forearm 73b of the left arm itself does not have a part that moves by a joint, and there is no joint between the worn wearable terminal 3 and the forearm 73b of the left arm. In contrast, the left hand 73c has finger joints, and therefore does not become a reference position for calculating the positional deviation. As another example, as shown in FIG. 2B, when the helmet 5 is worn on the head 71, the nose that is not covered by the helmet 5 is set in advance as a part to be a reference position. The nose itself does not have a part that moves by a joint, and there is no joint between the part covered by the helmet 5 and the nose. On the other hand, the jaw and mouth do not become the reference position for calculating the positional deviation because the lower jaw moves due to the jaw joint. As yet another example, as shown in FIG. 2C, when the camera 6 is attached to the right forearm 74b near the elbow, the right forearm 74b is set in advance as the part that becomes the reference position. The right forearm 74b itself does not have a part that moves due to a joint, and there is no joint between the attached camera 6 and the right forearm 74b (particularly the wrist side of the right forearm 74b). On the other hand, the right hand 74c moves due to the right wrist joint, so it does not become the reference position for calculating the positional deviation.

[0048] As an example, Figs. 6A to 6B show a method of calculating the positional deviation when the nose is identified as the reference position for calculating the positional deviation and the identified nose is set as the reference position. Fig. 6A is a plan view showing an example of an image acquired when the head 71 is photographed by the camera 6 shown in Fig. 2B. Fig. 6A includes the head 71 and the nose 711, and the nose 711 is set as the reference position for calculating the positional deviation. Here, the positional deviation in the case of Fig. 6A is calculated with respect to the position when the imaging device 11 is first put on, that is, the initial state position shown in Fig. 6A. In this case, if the helmet 5 is rotated counterclockwise, as shown in Fig. 6B, the head 71 appears to have rotated clockwise by an angle α with respect to the center 712 of the head 71 when the operator 7 is viewed in a plane, compared to the position of the imaging device 11 when the helmet 5 is first put on. However, if the helmet 5 is properly worn by the operator 7, the position of the nose 711 in the image does not change even if the operator 7 rotates. In reality, the operator 7 himself has not rotated, and only the helmet 5 has rotated. When an image is acquired in which the nose, which is the reference position, has moved relative to the initial position where the image capture device 11 is first attached, as shown in FIG. 6B, the calculation unit 23 detects the distance and angle (angle α in the case of FIG. 6B) by which the nose has moved relative to the position of the nose 711 in the initial state (i.e., the position of the center 712). Next, the calculation unit 23 calculates the positional deviation caused by the rotation of the helmet 5 (camera 6) from the detected distance and angle. Then, the calculation unit 23 outputs information on the calculated positional deviation to the correction unit 24. The initial position where the image capture device 11 is attached or the position of the nose 711 is also referred to as the starting position for calculating the positional deviation.

[0049] Alternatively, the part serving as the reference position may not be set in advance, but may be identified from the time-series images acquired by the acquisition unit 21. For example, in the time-series images captured at a predetermined cycle, a part that has moved only a distance within a predetermined range (for example, 5 mm or less) that allows a positional deviation of the mounting position of the imaging device 11 to be detected is detected, and the part is identified as the reference position. For example, when the imaging device 11 is mounted on the ankle of the right leg facing upward, a part that has moved an amount of 5 mm or less in the image is detected in the time-series images captured at a cycle of one minute. In this case, since the operator 7 is photographed from the ankle of the right leg facing upward, the calf of the right leg hardly moves relative to the ankle. Therefore, the calf of the right leg is identified as the reference position.

[0050] Furthermore, when the acquisition unit 21 acquires time-series images including a part serving as a reference position, the calculation unit 23 calculates the positional deviation from the amount of movement of the part in the time-series images. For example, the movement of the nose 711 shown in Fig. 6A and 6B can be acquired as time-series images from when the helmet 5 is put on until the helmet 5 rotates as shown in Fig. 6B, and the calculation unit 23 can calculate the direction and amount of movement of the nose 711 from when the helmet 5 is put on, and set the calculation result as the positional deviation. In this case, the starting position for calculating the positional deviation may be set to the first position of the part in the time-series images.

[0051] When calculating the positional displacement from the image acquired by the acquisition unit 21, the calculation unit 23 may use a positional displacement estimation model that has been trained in advance to estimate the positional displacement from the position of a part in the image that is set as a reference position. The positional displacement estimation model is stored, for example, in the database 14 shown in FIG. 1. In the positional displacement estimation model, an image including the operator 7 is associated with a positional displacement calculated in the image so as to correspond to the image, and when certain image data is input, a positional displacement corresponding to the input image data is output. The positional displacement estimation model is machine-learned by the second machine learning unit 231 shown in FIG. 1. Specifically, the positional displacement estimation model is trained to estimate an appropriate positional displacement for the input image data and output the result using past images and positional displacement calculation results stored in the database 14.

[0052] The positional deviation estimation model may be a trained model that has been subjected to machine learning. When the trained model receives the image data acquired by the acquisition unit 21 as input data, the trained model causes a computer (particularly, the CPU 131, which is a processor) to function so that output data including a positional deviation is output based on the input data. By using such a trained model, it is possible to calculate a positional deviation even for an operator 7 who performs an operation other than the trained operation.

[0053] The trained model of the present embodiment is not particularly limited, but may be, for example, a neural network. The neural network includes an input layer, an intermediate layer, and an output layer, and each layer includes at least one neuron. The input layer receives input data including image data acquired by the acquisition unit 21, and outputs the input data to the intermediate layer. The intermediate layer extracts data of the operator 7 from the data input from the input layer. Next, a positional deviation is estimated from the extracted data of the operator 7. The output layer outputs the data input from the intermediate layer as output data including data of the positional deviation. As parameters in the intermediate layer, the positions of the joints of the operator 7 and the connection relationship between the joints are considered.

[0054] Furthermore, when an image including a pattern for identifying the position of the imaging device 11 is acquired by the acquisition unit 21, the calculation unit 23 may identify the pattern as a reference position and calculate the positional deviation using the position of the imaging device 11 identified from the pattern. The pattern is not particularly limited as long as it serves as a reference when calculating the positional deviation in the image, and examples thereof include a plurality of intersecting lines and a star-shaped mark. For example, a grid line pattern is applied to a mask worn by a vehicle occupant, and the grid line is used to calculate the positional deviation of the wearing position of the imaging device 11. Furthermore, when the detection unit 22 detects the position of the imaging device 11 and the posture of the operator 7 using a detection device installed around the operator 7, the calculation unit 23 may calculate the positional deviation based on the position of the imaging device 11 and the posture of the operator 7 detected by the detection unit 22. For example, the position of the imaging device 11 and the posture of the operator 7 are detected using a camera installed in front of a worker in an assembly plant, and the behavior of the operator 7 is estimated using the detection result.

[0055] The correction unit 24 has a function of correcting the posture detected by the detection unit 22 using the positional deviation of the imaging device 11 calculated by the calculation unit 23. The correction unit 24 performs a geometric transformation on the posture of the operator 7, which is geometrically represented and acquired by the detection unit 22, using the information on the positional deviation calculated by the calculation unit 23. For example, as shown in FIG. 6B, when the helmet 5 rotates counterclockwise by an angle α, the correction unit 24 performs a transformation to rotate the detected posture clockwise by an angle α. This makes it possible to detect the posture of the operator 7 before the helmet 5 rotates and the wearing position is shifted, that is, the accurate posture of the operator 7. In addition, when the helmet 5 is shifted in the front-back direction and the left-right direction with respect to the operator 7, at least one linear transformation of translation, rotation, enlargement, and reduction is performed on the detected posture, and a process is performed to correct the movement (i.e., the positional deviation) of the helmet 5. The positional deviation can also be corrected for the wearable terminal 3 shown in FIG. 2A and the camera 6 shown in FIG. 2C by performing a similar geometric transformation.

[0056] The estimation unit 25 has a function of estimating the behavior of the operator 7 from the posture of the operator 7 corrected by the correction unit 24. The behavior of the operator 7 refers to any behavior required to complete a predetermined operation, and examples of the behavior include a behavior of a vehicle occupant reaching for a switch of an in-vehicle device such as an air conditioner to operate the in-vehicle device, a behavior of a worker at an assembly plant taking out a torque wrench from a toolbox, and a behavior of a mechanic at a dealer touching a switch for raising and lowering a lift that lifts a vehicle, as well as a behavior of a vehicle occupant pressing a switch for raising and lowering a window glass of the vehicle, a behavior of a vehicle occupant touching a touch panel to change the map display of a navigation device, a behavior of a worker at an assembly plant fitting a coupler connected to a sensor with a coupler connected to an electronic control unit (ECU), a behavior of a worker at an assembly plant tightening a bolt using a tool to attach an exhaust manifold to an engine block, a behavior of a mechanic at a dealer fitting a spark plug into an engine, and a behavior of a mechanic at a dealer tightening a bolt using a torque wrench.

[0057] For example, when the posture shown in Fig. 4A is detected, the estimation unit 25 estimates that the operator 7, who is a vehicle occupant, is reaching out his / her right hand (not shown) to an operation unit of the in-vehicle device to operate the in-vehicle device, based on the positions of the points P1-P3 and the shape of the straight line drawn from the points P1-P3 shown in Fig. 4A. Also, when the posture shown in Fig. 4B is detected, the estimation unit 25 estimates that the operator 7, who is a worker in the assembly plant, is trying to grasp a tool used for assembly with the left hand 73c and the right hand 74c to operate the tool, based on the positions of the points P7-P9 and P10-P12 shown in Fig. 4B and the shape of the straight line drawn from the points P7-P9 and P10-P12. Furthermore, when the posture shown in FIG. 4C is detected, it is estimated from the positions of points P19 to P32 shown in FIG. 4C and the shape of the straight line drawn from points P19 to P32 that the operator 7, who is a factory worker, is trying to grasp a tool used for assembly with his / her right hand 74c by extending his / her fingers. For example, pattern matching is used for estimating such behavior. Specifically, characteristic parts such as the positions of the joints of the operator 7, the relationship of the connections between the joints, and the shapes of the body parts connecting the joints are extracted from the input posture data, and the extracted characteristic parts are compared with the class dictionary stored in the database 14 to determine which class the characteristic parts belong to. If the behavior corresponding to the posture is set as a class in the class dictionary, it is possible to estimate the behavior from the posture.

[0058] When estimating a behavior, the estimation unit 25 can use a behavior estimation model that has been trained in advance to estimate the behavior of the operator 7 from the posture of the operator 7 detected by the detection unit 22 and corrected by the correction unit 24. The behavior estimation model is stored, for example, in the database 14 shown in FIG. 1. In the behavior estimation model, the detected posture of the operator 7 and the behavior of the operator 7 are associated so as to correspond to each other, and when data of a certain posture is input, the behavior of the operator 7 corresponding to the input posture data is output. The behavior estimation model is machine-learned by the third machine learning unit 251 shown in FIG. 1. Specifically, the behavior estimation model is trained to estimate an appropriate behavior for input posture data and output a result using past detection results of the posture of the operator 7 stored in the database 14 and the estimated behavior of the operator 7.

[0059] The behavior estimation model may be a trained model that has undergone machine learning. When posture data detected by the detection unit 22 is input as input data, the trained model causes a computer (particularly, the CPU 131, which is a processor) to function so that output data including the behavior of the operator 7 is output based on the input data. By using such a trained model, the behavior of the operator 7 performing an operation other than the trained operation can also be appropriately detected. Note that the input posture data may or may not be corrected by the correction unit 24.

[0060] The trained model of the present embodiment is not particularly limited, but may be, for example, a neural network. The neural network includes an input layer, an intermediate layer, and an output layer, and each layer includes at least one neuron. The input layer receives input data including posture data detected by the detection unit 22, and outputs the input data to the intermediate layer. The intermediate layer extracts posture data of the operator 7 from the data input from the input layer. Next, an action is estimated from the extracted posture data of the operator 7. The output layer outputs the data input from the intermediate layer as output data including action data. For example, the positions of points indicating the joints of the operator 7 and the shapes of straight lines indicating the connection relationships between the joints are considered as parameters in the intermediate layer.

[0061] Although multiple actions may be estimated for the posture of the operator 7, since the operator 7 in this embodiment performs a predetermined operation, the estimation unit 25 can identify the action corresponding to the posture if the attributes of the operator 7 are identified. The attributes of the operator 7 are characteristics regarding the role of the operator 7 and the location where the operator 7 is present, such as whether the operator 7 is a vehicle occupant, a worker working on an assembly line in an assembly plant, or a mechanic in a maintenance plant of a vehicle dealer.

[0062] The output unit 26 has a function of outputting the action estimated by the estimation unit 25 to the display device 12. The action of the operator 7 output from the output unit 26 is received by the display device 12. Then, it is displayed on the display device 12 and presented to the operator 7 and the supervisor of the operator 7. The operator 7 and the supervisor can confirm whether the action of the operator 7 is necessary for the predetermined operation, whether the operator 7 is performing the predetermined operation in a predetermined procedure, whether the operator 7 has skipped a necessary procedure, etc., by checking the action presented by the display device 12. Also, instead of the operator 7 and the supervisor, the processing unit 2 may determine whether the action of the operator 7 is necessary for the predetermined operation, whether the operator 7 is performing the predetermined operation in a predetermined procedure, whether the operator 7 has skipped a necessary procedure, etc. In this case, the predetermined operation, the procedure of the predetermined operation, the action necessary for the predetermined operation, etc. are stored in the database 14 and are acquired by the processing unit 2 as necessary.

[0063] [Processing in image processing systems] The procedure for processing information by the image processing device 13 will be described with reference to Fig. 7. Fig. 7 is an example of a flowchart showing information processing in the image processing system 1 of this embodiment. The process described below is executed at predetermined time intervals by the CPU 131, which is the processor of the image processing device 13, while the operator 7 is performing a predetermined operation.

[0064] First, in step S1, image data captured by the imaging device 11 is acquired by the function of the acquisition unit 21. In the following step S2, it is determined whether image data including the operator 7 has been acquired. If the acquired image data does not include the operator 7, the process returns to step S1. On the other hand, if the acquired image data includes the operator 7, the process proceeds to step S3.

[0065] In step S3, the posture of the operator 7 included in the image is detected by the function of the detection unit 22. In the following step S4, it is determined whether or not the posture of the operator 7 has been properly detected. If the posture of the operator 7 has not been properly detected, the process returns to step S1. On the other hand, if the posture of the operator 7 has been properly detected, the process proceeds to step S5.

[0066] In step S5, the function of the calculation unit 23 is used to calculate how much the mounting position of the imaging device 11 is displaced from, for example, a reference position in the initial state. In the following step S6, it is determined whether the mounting position of the imaging device 11 is displaced. Specifically, it is determined whether the calculated amount of positional deviation is within a range in which the posture of the operator 7 can be correctly detected. If the calculated amount of positional deviation is within the range in which the posture of the operator 7 can be correctly detected, the process proceeds to step S8. On the other hand, if the calculated amount of positional deviation exceeds the range in which the posture of the operator 7 can be correctly detected, the process proceeds to step S7.

[0067] In step S7, the posture of the operator 7 is corrected based on the deviation calculated by the calculation unit 23 using the function of the correction unit 24. In the following step S8, the behavior of the operator 7 is estimated from the posture of the operator 7 using the function of the estimation unit 25. In the following step S9, it is determined whether the behavior of the operator 7 has been appropriately estimated. If the behavior of the operator 7 has not been appropriately estimated, execution of the routine is stopped and information processing is terminated. On the other hand, if the behavior of the operator 7 has been appropriately estimated, the process proceeds to step S10.

[0068] In step S10, the output unit 26 functions to output data including the estimated behavior from the image processing device 13 to the display device 12. In the following step S11, the processing unit 2 functions to determine whether or not the operator 7 is properly performing a predetermined operation based on the behavior estimated by the estimation unit 25. At this time, the determination result may be output to the display device 12. When the processing in step S11 is completed, the execution of the routine is stopped and the processing of the information is terminated. Note that steps S2, S4, S6, and S9 to S11 are not essential steps and can be provided as necessary.

[0069] [Embodiments of the present invention] As described above, according to this embodiment, there is provided an image processing device 13 including an acquisition unit 21 that acquires an image including the operator 7 performing a predetermined operation from an imaging device 11 attached to at least one of the operator 7 performing a predetermined operation and an article worn by the operator 7, a detection unit 22 that detects the posture of the operator 7 from the image, a calculation unit 23 that calculates a positional deviation of the imaging device 11 from the image using a reference position as a reference for calculating the positional deviation, and a correction unit 24 that corrects the posture using the positional deviation. As a result, even if the mounting position of the imaging device 11 (for example, a camera) is shifted, the posture of the operator 7 can be correctly recognized.

[0070] Moreover, the image processing device 13 of this embodiment includes an estimation unit 25 that estimates the behavior of the operator 7 from the corrected posture by using a behavior estimation model that has been trained in advance to estimate the behavior of the operator 7 from the posture. This makes it possible to associate the posture and behavior of the operator 7 and estimate the behavior of the operator 7.

[0071] Moreover, according to the image processing device 13 of this embodiment, the detection unit 22 detects the posture of the operator 7 from the image using a posture estimation model that has been trained in advance to estimate the posture of the operator 7 from the image. This makes it possible to perform posture detection with high accuracy using detection results obtained up to now.

[0072] Moreover, according to the image processing device 13 of this embodiment, the calculation unit 23 identifies a part of the body of the operator 7 included in the image as the reference position, and calculates the positional deviation from the reference position in a predetermined state. This makes it possible to realize simple and accurate calculation of the positional deviation using the body part.

[0073] Moreover, according to the image processing device 13 of this embodiment, when the imaging device 11 is attached to the operator, the reference position is a part of the body of the operator 7 other than the part where the imaging device 11 is attached that does not have a part that can move by a joint and does not have a joint between the part where the imaging device 11 is attached, and when the imaging device 11 is attached to the article, the reference position is a part of the body of the operator 7 other than the part in contact with the article that does not have a part that can move by a joint and does not have a joint between the part in contact with the article worn by the operator 7. This makes it possible to identify the body parts more accurately.

[0074] Moreover, according to the image processing device 13 of this embodiment, the acquisition unit 21 acquires time-series images including the reference position, and the calculation unit 23 calculates the positional deviation using the amount of movement of the reference position in the time-series images, thereby making it possible to more accurately calculate the positional deviation of the imaging device 11.

[0075] According to the image processing device 13 of this embodiment, the calculation unit 23 calculates the positional deviation using a positional deviation estimation model that is pre-trained to estimate the positional deviation from the position of the reference position in the image, thereby making it possible to calculate the amount of positional deviation by simply inputting an image.

[0076] Furthermore, according to the image processing device 13 of this embodiment, the acquisition unit 21 acquires an image including a pattern for identifying the position of the imaging device 11, and the calculation unit 23 identifies the pattern as the reference position and calculates the positional deviation using the position of the imaging device 11 identified from the pattern. This makes it possible to more accurately calculate the positional deviation of the imaging device 11.

[0077] Furthermore, according to the image processing device 13 of this embodiment, the detection unit 22 detects the position of the imaging device 11 and the posture of the operator 7 using a detection device installed around the operator 7, and the calculation unit 23 calculates the positional deviation based on the position of the imaging device 11 and the posture of the operator 7 detected by the detection unit 22. This makes it possible to more accurately calculate the positional deviation of the imaging device 11.

[0078] Also, according to this embodiment, there is provided an image processing method executed by a processor, in which the processor acquires an image including the operator 7 performing a predetermined operation from an image capture device 11 attached to at least one of the operator 7 performing the predetermined operation and an article worn by the operator 7, detects the posture of the operator 7 from the image, calculates a positional deviation of the image capture device 11 from the image using a reference position as a reference for calculating the positional deviation, and corrects the posture using the positional deviation. As a result, even if the mounting position of the image capture device 11 (e.g., a camera) is shifted, the posture of the operator 7 can be correctly recognized. [Explanation of symbols]

[0079] 1. Image processing system 11...Imaging device 12...Display device 13...Image processing device 131...CPU 132...ROM 133…RAM 14. Database 2. Processing section 21…Acquisition part 22…Detection unit 221…1st Machine Learning Department 23...Calculation section 231…2nd Machine Learning Department 24…Compensation section 25…Estimation part 251…Third Machine Learning Department 26...Output section 3. Wearable devices 31…Band 4, 4a, 4b...field of view 5. Helmet 51…Brim 52…Bracket 6. Camera 61…Band 7…Operator (human) 71...Head 711...nose 712…center 72...Neck 73a…Left upper arm 73b…Left forearm 73c…Left hand 74a…Right upper arm 74b…Right forearm 74c…Right hand 75…Torso 76...Left leg 77...Right leg 8, 9…Arrows P1~P32…points α…Angle

Claims

1. an acquisition unit that acquires an image including the human from an imaging device attached to at least one of a human and an item worn by the human; a detection unit for detecting a posture of the person from the image; a calculation unit that calculates a positional deviation of the imaging device from the image by using a reference position as a reference for calculating the positional deviation; a correction unit that corrects the attitude using the positional deviation.

2. The image processing device according to claim 1 , further comprising an estimation unit that estimates the behavior of the human being from the corrected posture by using a behavior estimation model that has been trained in advance to estimate the behavior of the human being from the posture.

3. The image processing device according to claim 1 , wherein the detection unit detects the posture of the human from the image using a posture estimation model that has been trained in advance to estimate the posture of the human from the image.

4. 4. The image processing device according to claim 1, wherein the calculation unit identifies a part of the human body included in the image as the reference position, and calculates the positional deviation with respect to the reference position in a predetermined state.

5. The reference position is When the imaging device is attached to the human, among the parts of the body of the human other than the part to which the imaging device is attached, the part itself does not have a part that can move by a joint, and there is no joint between the part to which the imaging device is attached and the part, 5. The image processing device according to claim 4, wherein, when the imaging device is attached to the article, among the parts of the human body other than the part in contact with the article, the part itself does not have a part that can move by a joint, and there is no joint between the part in contact with the article worn by the human and the part.

6. The acquisition unit acquires a time series of images including the reference position, The image processing device according to claim 4 , wherein the calculation unit calculates the positional deviation using an amount of movement of the reference position in the time-series images.

7. The image processing device according to claim 4 , wherein the calculation unit calculates the positional deviation using a positional deviation estimation model that is trained in advance to estimate the positional deviation from the position of the reference position in the image.

8. the acquisition unit acquires an image including a pattern for identifying a position of the imaging device; The image processing device according to claim 1 , wherein the calculation section identifies the pattern as the reference position, and calculates the positional deviation using a position of the imaging device identified from the pattern.

9. An acquisition unit that acquires an image including the human from an imaging device attached to at least one of a human and an item worn by the human; a detection unit that detects a posture of the person from the image acquired by the acquisition unit from the imaging device, and further detects a position and the posture of the imaging device using a detection device installed around the person; a calculation unit that calculates a positional deviation of the imaging device using a reference position as a reference for calculating a positional deviation based on the position and the orientation of the imaging device detected by the detection device; a correction unit that corrects the orientation using the positional deviation calculated by the calculation unit when the detection unit detects the orientation from the image.

10. 1. A processor-implemented method for image processing, comprising: The processor, acquiring an image including the human from an imaging device attached to at least one of the human and an item worn by the human; Detecting a pose of the person from the image; calculating a positional deviation of the imaging device from the image using a reference position as a reference for calculating the positional deviation; The positional deviation is used to correct the attitude.

Images