Image processing device and image processing method

Abstract

The present invention provides an image processing device (13) and an image processing method, wherein the image processing device (13) obtains an image containing the person (7) from a camera device (11) worn on at least one of a person (7) and an item carried by the person (7), detects the posture of the person (7) from the image, calculates the position deviation of the camera device (11) based on the image using a reference position as a reference for calculating the position deviation, and uses the position deviation to correct the posture.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method for identifying human behavior. Background Art

[0002] The following method is known: obtaining a pair of three-dimensional image data and two-dimensional image data captured by a camera from the photographer's side, estimating the position and posture of the camera based on the three-dimensional image data, extracting skeleton posture parameters representing the skeleton posture of the photographer based on the estimated position and posture of the camera, body parameters representing the body shape of the photographer calculated in advance, and the three-dimensional image data, extracting posture feature quantities of the photographer based on the skeleton posture parameters, extracting image feature quantities based on the two-dimensional image data, and identifying the photographer's actions based on the posture feature quantities and image feature quantities and identifier parameters obtained by pre-learning for identifying the photographer's actions (Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2016-99982 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, the above-mentioned conventional technology has a problem that, when the wearing position of the camera mounted on the head of a person or the like is deviated from the original position, the posture of the person cannot be accurately identified using the image data acquired by the camera.

[0008] The problem to be solved by the present invention is to provide an image processing device and an image processing method which can correctly identify a person's posture even when the wearing position of a camera is deviated.

[0009] Solutions for solving problems

[0010] The present invention solves the above-mentioned problem through the following processing: acquiring an image containing the person from a camera device worn on at least one of the person and an item carried by the person; detecting the posture of the person from the image; calculating the position deviation of the camera device based on the image using a reference position as a reference for calculating the position deviation; and correcting the posture using the position deviation.

[0011] Effects of the Invention

[0012] According to the present invention, even when the wearing position of the camera is deviated, the posture of a person can be accurately identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a block diagram showing one embodiment of an image processing system including the image processing device according to the present invention.

[0014] Figure 2A It shows that Figure 1 A top view of the wearable terminal of the camera device and the operator's left forearm and left hand.

[0015] Figure 2B The figure shows the head of a person (operator) and the Figure 1 A stereoscopic view of a camera helmet with a camera device.

[0016] Figure 2C is shown as Figure 1 A top view of the camera of the video device and the operator's right arm.

[0017] Figure 3A It is shown by Figure 2A An example of an image of an operator captured by a wearable terminal.

[0018] Figure 3B It is shown by Figure 2B An example of an image of an operator captured by a camera.

[0019] Figure 3C It is shown by Figure 2C An example of an image of an operator captured by a camera.

[0020] Figure 4A is shown for detection Figure 3A FIG. 1 is a diagram showing an example of a method of an operator's posture.

[0021] Figure 4B is shown for detection Figure 3B FIG. 1 is a diagram showing an example of a method of an operator's posture.

[0022] Figure 4C is shown for detection Figure 3C FIG. 1 is a diagram showing an example of a method of an operator's posture.

[0023] Figure 5A It is shown by Figure 2A FIG. 1 is another example of an image of an operator captured by a wearable terminal.

[0024] Figure 5B It is shown by Figure 2C FIG. 1 is another example of an image of an operator captured by a camera.

[0025] Fig. 6A It is shown by Figure 2BFIG. 1 is another example of an image of an operator captured by a camera.

[0026] Figure 6B It is shown by Figure 2B Figure 2 is another example of an image of an operator captured by a camera.

[0027] Figure 7 It is shown Figure 1 A flowchart of an example of a process of information processing in an image processing system. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of an image processing device and an image processing method according to the present invention will be described with reference to the drawings.

[0029] [Image processing system]

[0030] Figure 1 1 is a block diagram showing an image processing system 1 according to the present invention. The image processing system 1 is a device for detecting the action of an operator (equivalent to the person according to the present invention) who performs a predetermined operation. The predetermined operation is not particularly limited, and examples thereof include the operation of an onboard device by a vehicle occupant (including a driver, the same applies hereinafter) , the operation of a tool by an operator engaged in assembly work in a factory, and the operation of a maintenance tool by a maintenance worker engaged in vehicle maintenance at a vehicle sales store (hereinafter also referred to as a "dealer" ). The operator whose action is detected by the image processing system 1 (hereinafter also referred to as an "operator") is not particularly limited, and examples thereof include a vehicle occupant, a factory operator, and a maintenance worker at a dealer. As for the action detected by the image processing system 1, examples thereof include the action of a vehicle occupant reaching for a switch of an onboard device in order to operate an onboard device such as an air conditioner, the action of an operator at an assembly plant taking out a torque wrench from a tool box, and the action of a maintenance worker at a dealer touching a switch for raising or lowering a lift for lifting a vehicle. By using the image processing system 1 of the present embodiment to detect the above-mentioned actions, it is possible to confirm whether the functions associated with the operation of the vehicle-mounted equipment are properly discovered, whether the assembly work of the workers engaged in the assembly work in the vehicle assembly factory is carried out according to the predetermined process, whether the maintenance of the maintenance personnel engaged in the maintenance of the vehicle at the vehicle sales store is carried out according to the instructions, etc.

[0031] like Figure 1As shown, the image processing system 1 includes a camera 11, a display device 12, and an image processing device 13. The devices constituting the image processing system 1 are connected by a known method such as a wired or wireless LAN in a state where data can be exchanged with each other. The number of the camera 11 and the display device 12 is not particularly limited as long as there is at least one each. In addition, the camera 11 and the display device 12 do not need to be set together with the image processing device 13, and may be set at a location away from the image processing device 13. For example, the camera 11 and the display device 12 may be set near the assembly line of an assembly plant, and the image processing device 13 may be set in a central control room away from the assembly line or in a server far away from the assembly plant.

[0032] The camera device 11 is a device for acquiring image data of objects existing around the operator, and is, for example, a camera having an imaging element such as a CCD, an ultrasonic camera, an infrared camera, or the like. In addition to the operator, the object also includes objects existing around the operator. For example, switches and touch panels around the occupants of the vehicle, accessories installed by the operator, tools used, and vehicles repaired by the maintenance personnel of the dealer are included in the object. In addition, the camera device 11 is set at a position where the operator's body posture can be detected, such as the dashboard, roof and seat of the vehicle, the assembly line of the assembly plant, the workbench and the tools used by the operator, and the dealer's lift.

[0033] The camera device 11 of this embodiment is worn on at least one of an operator who performs the above-mentioned prescribed operation and an item carried by the operator. At this time, the camera device 11 is installed so as to be able to capture at least a part of the operator's body. When the camera device 11 is worn on the operator, for example, the camera device 11 is worn on the operator's head, upper arm, forearm, abdomen, thigh, calf, etc. As a wearing method, for example, a method of fixing it by wrapping it around the operator's body using Velcro or a buckle, a method of sticking it to the operator's body using an adhesive tape, and a method of sticking it to the operator's body using an adhesive.

[0034] As an example, when the camera device 11, the display device 12 and the image processing device 13 are combined into a wearable terminal 3 such as a smart watch, Figure 2A Wear as shown. Figure 2A 7 is a top view showing the forearm 73b and left hand 73c of the operator's left arm, and the forearm 73b of the left arm is covered by clothes. In this case, the wearable terminal 3 is wrapped around the part of the forearm 73b of the operator's left arm near the wrist by a band 31 with Velcro. The wearable terminal 3 has a camera 11, which captures Figure 2A4. Thus, an image including the body of the operator wearing the wearable terminal 3 can be captured.

[0035] On the other hand, the articles carried by the operator refer to the clothes and protective gear worn by the operator when performing the prescribed operation. Examples of the clothes and protective gear include jackets, coats, watches, rings, earrings and earrings worn by the occupants of the vehicle, and work clothes, hats, safety shoes, helmets, masks and protective glasses worn by the operators of the assembly plant and the maintenance personnel of the dealer. As a method of wearing the camera device 11 on the clothes and protective gear worn by the operator, for example, there can be cited a method of fixing it to a coat or work clothes using Velcro, a method of wearing it on a hat, helmet, mask, protective glasses, etc. by installing a member such as a clamp on the hat, helmet, mask, protective glasses, etc. and holding the camera device 11 with the member, and a method of bonding it to a watch or safety shoes using an adhesive tape or adhesive.

[0036] As an example, when the camera device 11 is a CCD camera with a wide-angle lens, a fisheye lens, or an ultra-wide-angle lens, Figure 2B Wear as shown. Figure 2B : is a stereoscopic diagram showing the head 71 of the operator and the helmet 5 worn on the head 71. In this case, the camera 6 is mounted downward on the brim 51 of the helmet 5 worn by the operator so as to be able to shoot the body of the operator. In the mounting method of the camera 6, a bracket 52 for holding the camera 6 is mounted on the helmet 5, and the camera 6 is held by the bracket 52. Alternatively, a portion for clamping the camera 6 may be provided on the helmet 5 and the portion may be integrally formed as the helmet 5 to clamp the camera 6. The camera 6 shoots Figure 2B In this way, an image including the body of the operator wearing the helmet 5 can be captured.

[0037] In addition, when the camera 6 is worn by the operator, for example Figure 2C Wear as shown. Figure 2C 7 is a top view showing the operator's right arm, which is composed of the upper arm 74a, the forearm 74b and the right hand 74c. In this case, the camera 6 is wrapped around the portion of the forearm 74b of the operator's right arm near the elbow by a band 61 having Velcro. The camera 6 captures Figure 2C The image of the range of the field of view 4b shown by the dotted line in FIG. 6 can thereby capture an image including the right hand 74c of the operator wearing the camera 6. Figure 2CIn the embodiment, the camera 6 is worn so as to be able to photograph the back side of the operator's right hand 74c, but the camera 6 may be worn so as to be able to photograph the palm side of the operator's right hand 74c. Which side is photographed is appropriately selected according to the detected action of the operator. Figure 2A In the example, the wearable terminal 3 is worn on the left arm. Figure 2C In the example of , the camera 6 is worn on the right arm, but the wearable terminal 3 and the camera 6 may be worn on either the left or right arm. In addition, these wearing positions can be set at appropriate positions within the range in which the action of the operator can be appropriately detected.

[0038] Return to Figure 1 The display device 12 is a device for notifying the operator of the action detected by the image processing device 13. The display device 12 is, for example, a liquid crystal display, a projector, etc., and may also be provided with a speaker. The display device 12 is arranged at a location such as the dashboard of a vehicle, the workplace of an operator in an assembly plant, etc., which is close to the operator and can notify the operator of the required information. However, in the case where there is a supervisor who monitors the operator's operation, the display device 12 is arranged near the supervisor. In this case, if the supervisor is located at a place far away from the operator, the display device 12 is arranged at a location far away from the operator. In addition, the display device 12 may also be installed on the operator as a wearable terminal such as glasses. Furthermore, the display device 12 may be replaced by a speaker that only emits an alarm sound according to the detected action.

[0039] The image processing device 13 is a device for estimating the behavior of the operator based on the posture of the operator performing a predetermined operation, and is, for example, a computer. In particular, the image processing device 13 can take into account the position deviation of the camera 11 when detecting the posture of the operator. The image processing device 13 acquires image data from the camera 11 at predetermined time intervals by cooperating with the camera 11 and the display device 12, and performs detection of the operator's posture, correction of the position deviation of the camera 11, and estimation of the operator's behavior by processing the acquired image data.

[0040] In order to realize these data acquisition functions, posture detection function, position deviation correction function, and behavior estimation function, the image processing device 13 includes a CPU (Central Processing Unit) 131 as a processor, a ROM (Read Only Memory) 132 storing programs, and a RAM (Random Access Memory) 133 functioning as an accessible storage device. The CPU 131 is an operation circuit for functioning as the image processing device 13 by executing the program stored in the ROM 132. In addition, the image processing device 13 does not need to be provided together with the camera device 11 and the display device 12, and can also be provided in a server at a distance away from these devices.

[0041] [Processing Department]

[0042] The program used in the image processing device 13 includes a processing unit 2, which is a functional block for realizing the functions of the above-mentioned image processing device 13. The processing unit 2 controls the camera device 11 and the display device 12 to make them cooperate, thereby realizing a function of acquiring image data containing the operator from the camera device 11 (data acquisition function), a function of detecting the operator's posture after processing the acquired image data and correcting the position deviation of the camera device 11 (posture detection function and position deviation correction function), and a function of estimating the operator's action based on the operator's posture (action estimation function). As functional blocks corresponding to these functions, such as Figure 1 As shown in FIG. 1 , the processing unit 2 includes an acquisition unit 21, a detection unit 22, a calculation unit 23, a correction unit 24, an estimation unit 25, and an output unit 26. Figure 1 In the figure, each part is extracted and shown for convenience.

[0043] Figure 1 The image processing device 13 shown in the figure has all the above-mentioned functional blocks, but a single image processing device 13 does not need to have all the functional blocks, and some of the above-mentioned functional blocks can also be set in other devices included in the image processing system 1, or in another information processing device not shown in the figure. Figure 1 In the image processing system 1, the detection unit 22 may be provided in the imaging device 11. In this case, the function of the detection unit 22 is executed using the CPU, ROM, and RAM of the imaging device 11.

[0044] In addition, it is not necessary for a single device to execute all the processing of each functional block, and the functions of each functional block can be implemented across multiple devices connected in a state where data can be exchanged. Figure 1In the image processing system 1, part of the processing performed by the calculation unit 23 is performed by the imaging device 11, and the rest of the processing is performed by the image processing device 13. In this case, part of the processing for realizing the function of the calculation unit 23 is performed using the CPU, ROM, and RAM of the imaging device 11.

[0045] Next, the functions realized by each functional block of the processing unit 2 will be described.

[0046] The acquisition unit 21 has a function of acquiring an image including an operator from the imaging device 11. Figure 2A to Figure 2C As shown, the camera device 11 of this embodiment is set at a position and orientation that can capture an image containing the operator's body, so the acquisition unit 21 can acquire the image containing the operator by acquiring the image data captured by the camera device 11 at a predetermined time interval. In addition, the image data acquired by the acquisition unit 21 is not limited to still images, but can also be a time-series image such as a moving image. Figure 3A to Figure 3C An example of an image acquired by the acquisition unit 21 is shown in FIG.

[0047] Figure 3A Is Figure 2A The image captured by the wearable terminal 3 shown in FIG. 1 is an image including the operator 7 as a passenger of the vehicle. The operator 7 as a passenger of the vehicle wears the wearable terminal 3 on the forearm 73b of the left arm, so when the operator 7 is captured from the wearable terminal 3, as shown in FIG. Figure 3A As shown, the upper arm 73a of the left arm and the forearm 73b of the left arm are shown on the right side, and the upper arm 74a of the right arm and the forearm 74b of the right arm are shown on the left side. In addition, the entire head 71 and neck 72 of the operator 7 are photographed, and a part of the torso 75 is also photographed.

[0048] Figure 3B Is Figure 2B The image captured by the camera 6 shown in the figure includes an image of the operator 7 who is a worker in the vehicle assembly plant. The operator 7 who is a worker wears the camera 6 on the brim 51 of the helmet 5 worn on the head 71 with the camera 6 facing downward. Therefore, when the operator 7 is photographed from the camera 6, the part of the head 71 above the brim 51 (that is, the part covered by the helmet 5) cannot be photographed. Figure 3B As shown in FIG. 6 , the image cannot be obtained for the upper part of the head 71. The neck 72 and the torso 75 are displayed below the head 71, the upper arm 74a of the right arm, the forearm 74b of the right arm, and the right hand 74c are displayed on the left side of the torso 75, and the upper arm 73a of the left arm, the forearm 73b of the left arm, and the left hand 73c are displayed on the right side of the torso 75. Since the camera 6 shoots the operator 7 from the front direction of the operator 7, the camera 6 is not directly connected to the operator 7. Figure 3ALikewise, the left and right are reversed. In addition, a part of the left leg 76 and a part of the right leg 77 are photographed on the lower side of the trunk 75.

[0049] Figure 3C Is Figure 2C The image captured by the camera 6 shown in FIG. 1 includes the right hand 74c of the operator who is a worker in the vehicle assembly factory. The operator who is the worker wears the camera 6 on the part of the forearm 74b of the right arm near the elbow. Therefore, when the right hand 74c is captured by the camera 6, as shown in FIG. Figure 3C As shown, the wrist side of the forearm 74b of the right arm and the right hand 74c are shown.

[0050] Return to Figure 1 , the detection unit 22 has a function of detecting the posture of the operator 7 based on the image acquired by the acquisition unit 21. The posture of the operator 7 refers to the posture or attitude of the body of the operator 7, specifically, the position of each joint of the body of the operator 7 and the relationship between the joints. For example, as long as the position of each joint of the body of the operator 7 and the parts of the body connecting the joints are obvious, the posture of the operator's body can be geometrically represented by drawing points at coordinates corresponding to the positions of each joint in a three-dimensional space defined in an orthogonal coordinate system, and connecting the joints (points) connected by the parts with straight lines.

[0051] As an example, when the acquisition unit 21 acquires Figure 3A In the case of the image shown in FIG. Figure 4A The posture of the operator 7 is detected by performing the processing shown in FIG. Figure 4A In the present invention, first, the joints of the operator 7 are detected from the operator 7 included in the image by pattern matching or the like. Specifically, characteristic parts such as the parts where the orientation of the operator 7's body changes are extracted from the input image data, and the extracted characteristic parts are compared with the class dictionary stored in the database 14, thereby determining which class the characteristic parts belong to. If the class of each joint is set in the class dictionary, it is possible to determine whether the characteristic parts correspond to the joints.

[0052] Next, points are drawn on the parts determined to be joints. Specifically, Figure 4AAs shown, point P1 is drawn at the part corresponding to the joint of the neck 72 of the operator 7, point P2 is drawn at the part corresponding to the joint of the right shoulder, point P3 is drawn at the part corresponding to the joint of the right elbow, point P4 is drawn at the part corresponding to the joint of the left shoulder, and point P5 is drawn at the part corresponding to the joint of the left elbow. Next, a straight line corresponding to the right shoulder is drawn between point P1 and point P2, a straight line corresponding to the upper arm 74a of the right arm is drawn between point P2 and point P3, a straight line corresponding to the left shoulder is drawn between point P1 and point P4, and a straight line corresponding to the upper arm 73a of the left arm is drawn between point P4 and point P5.

[0053] In addition, Figure 3A The image shown does not include the right hand and left hand of the operator 7, but since there is no joint between the elbow joint and the wrist, it is estimated that the right hand is present. Figure 4A The left hand is located on the extension line of the forearm 74b of the right arm included in the figure, and the left hand is located on the extension line of the forearm 73b of the left arm. Figure 4A As shown, a straight line is drawn from point P3 toward the estimated position of the right hand (not shown), and a straight line is drawn from point P5 toward the estimated position of the left hand (not shown). These straight lines, for example, pass through the center of the forearm 74b of the right arm and the forearm 73b of the left arm in a straight direction along the length direction of the forearm 74b of the right arm and the forearm 73b of the left arm. Figure 3A The image shown does not include the part of the body 75 below the waist, but a straight line is drawn from the point P1 corresponding to the joint of the neck 72 through the left and right center parts of the body 75 to indicate the position of the body 75. Figure 4A The points P1 to P5 shown and the straight lines drawn from each point represent Figure 3A The positions of the joints of the body of the operator 7 included in the image and the relationship between the joints are detected, thereby detecting the posture of the operator 7.

[0054] In addition, as another example, after the acquisition unit 21 acquires Figure 3B In the case of the image shown in FIG. Figure 4B The posture of the operator 7 is detected by performing the processing shown in FIG. Figure 4B First, by Figure 4A The joints are detected by the same pattern matching process as described in . Figure 4A Similarly, points are drawn on the parts judged to be joints. Specifically, Figure 4BAs shown, at the partial depiction point P6 corresponding to the joint of the neck 72 of the operator 7, at the partial depiction point P7 corresponding to the joint of the right shoulder, at the partial depiction point P8 corresponding to the joint of the right elbow, at the partial depiction point P9 corresponding to the joint of the right wrist, at the partial depiction point P10 corresponding to the joint of the left shoulder, at the partial depiction point P11 corresponding to the joint of the left elbow, at the partial depiction point P12 corresponding to the joint of the left wrist. In addition, at the partial depiction point P13 corresponding to the joint of the waist, at the partial depiction point P14 corresponding to the joint of the right thigh root, at the partial depiction point P15 corresponding to the joint of the right knee, at the partial depiction point P16 corresponding to the joint of the left thigh root, at the partial depiction point P17 corresponding to the joint of the left knee.

[0055] Next, a straight line corresponding to the right shoulder is drawn between point P6 and point P7, a straight line corresponding to the upper arm 74a of the right arm is drawn between point P7 and point P8, a straight line corresponding to the forearm 74b of the right arm is drawn between point P8 and point P9, a straight line corresponding to the left shoulder is drawn between point P7 and point P10, a straight line corresponding to the upper arm 73a of the left arm is drawn between point P10 and point P11, and a straight line corresponding to the forearm 73b of the left arm is drawn between point P11 and point P12. Furthermore, a straight line corresponding to the trunk 75 is drawn between point P6 and point P13, a straight line corresponding to the heel is drawn between point P13 and point P14 and between point P13 and point P16, a straight line corresponding to the right leg 77 is drawn between point P14 and point P15, and a straight line corresponding to the left leg 76 is drawn between point P16 and point P17.

[0056] In addition, Figure 3B The image shown does not include the right ankle and the left ankle of the operator 7, but since there is no joint between the knee joint and the ankle, it is estimated that the right ankle is present. Figure 4B The left ankle is located on the extension line of the left leg 76. Figure 4B As shown in FIG. 1 , straight lines are drawn from point P15 and point P17 toward the estimated positions of the two ankles (not shown). These straight lines, for example, pass through the centers of the right leg 77 and the left leg 76 in a direction straight along the length direction of the right leg 77 and the left leg 76. In this way, the detection unit 22 Figure 4B The points P6 to P17 and the straight lines drawn from the points are shown. Figure 3B The positions of the joints of the body of the operator 7 included in the image and the relationship between the joints are detected, thereby detecting the posture of the operator 7.

[0057] And, as another example, after the acquisition unit 21 acquires Figure 3C In the case of the image shown in FIG. Figure 4C The processing shown in FIG. 7 is used to detect the posture of the operator's right hand 74c. Figure 4C First, by Figure 4A The joints are detected by the same pattern matching process as described in . Figure 4A Similarly, points are drawn on the parts judged to be joints. Specifically, Figure 4C As shown, point P18 is drawn on the part corresponding to the joint of the operator's right wrist, points P19 to P20 are drawn on the part corresponding to the joint of the thumb, points P21 to P23 are drawn on the part corresponding to the joint of the index finger, points P24 to P26 are drawn on the part corresponding to the joint of the middle finger, points P27 to P29 are drawn on the part corresponding to the joint of the ring finger, and points P30 to P32 are drawn on the part corresponding to the joint of the little finger. Then, straight lines corresponding to the palm are drawn between point P18 and point P19, between point P18 and point P21, between point P18 and point P24, between point P18 and point P27, and between point P18 and point P30. Furthermore, a straight line corresponding to the thumb is drawn between point P19 and point P20, a straight line corresponding to the index finger is drawn between point P21 and point P22 and between point P22 and point P23, a straight line corresponding to the middle finger is drawn between point P24 and point P25 and between point P25 and point P26, a straight line corresponding to the ring finger is drawn between point P27 and point P28 and between point P28 and point P29, and a straight line corresponding to the little finger is drawn between point P30 and point P31 and between point P31 and point P32.

[0058] In addition, Figure 3C The image shown does not include the operator's right elbow, but since there is no joint between the elbow and the wrist, it is assumed that the right elbow is present. Figure 4C The forearm 74b of the right arm is included in the extension line. Figure 4C As shown in FIG. 1 , a straight line is drawn from point P18 toward the estimated position of the right elbow (not shown). This straight line passes through the center of the forearm 74b of the right arm in a direction straight along the length direction of the forearm 74b of the right arm. In this way, the detection unit 22 passes Figure 4C The points P18 to P32 shown in the figure and the straight lines drawn from each point represent Figure 3C The positions of the joints of the operator's right hand 74 c and the relationship between the joints included in are used to detect the posture of the operator's right hand 74 c.

[0059] In addition to the above-mentioned method, the detection unit 22 can also detect the posture of the operator 7 based on the relative positional relationship between the parts of the operator 7's body and the objects around the operator 7. For example, the position of the head 71 and arms of the operator 7 relative to the ground on which the operator 7 stands is calculated, and the posture of the operator 7 is detected based on the result of the calculation. In addition, instead of or in addition to this, the detection unit 22 can also detect the posture based on the orientation of the parts of the operator 7's body. For example, the posture of the operator 7 is detected based on the position of the operator 7's eyes (line of sight), the orientation of the hands, the orientation of the feet, and the orientation of the torso 75.

[0060] In addition, the joints to which the detection unit 22 sets the points are not limited to Figure 4A to Figure 4C The joints shown. Figure 4A and Figure 4B Medium detection ratio Figure 4A and Figure 4B The joints shown are thinner and more points are depicted. For example, Figure 4C As shown in the figure, points are drawn at the parts corresponding to the joints of the fingers of the left hand 73c and the right hand 74c to detect the postures of the fingers of the left hand 73c and the right hand 74c. In this way, it is possible to detect subtle postures such as whether the vehicle occupant is touching the vehicle's onboard equipment or whether the assembly line worker is holding a tool.

[0061] When detecting the posture of the operator 7 from the image acquired by the acquisition unit 21, the detection unit 22 may use a posture estimation model that has been previously learned so as to estimate the posture of the operator 7 from the image including the operator 7. The posture estimation model is stored in, for example, Figure 1 The posture estimation model includes an image of the operator 7 and a posture of the operator 7 detected in the image. When a certain image data is input, the posture of the operator 7 corresponding to the input image data is output. The posture estimation model is based on Figure 1 The first machine learning unit 221 shown here performs machine learning. Specifically, the posture estimation model is trained using past images stored in the database 14 and the detection results of the posture of the operator 7 so as to detect an appropriate posture for the input image data and output the result.

[0062] In addition, the posture estimation model may also be a learned model obtained by machine learning. A learned model is a model that is learned in advance by machine learning in a manner that obtains appropriate output data for certain input data, and at least has a program for performing operations until output data is obtained based on the input data, and a weighting coefficient (parameter) for the operation. The learned model enables the computer (particularly the CPU 131 as a processor) to function in a manner that, when image data acquired by the acquisition unit 21 is input as input data, outputs output data including the posture of the operator 7 based on the input data. By using such a learned model, the posture can be detected even for the operator 7 who performs an operation other than the learned operation.

[0063] The learning model of the present embodiment is not particularly limited, but is, for example, a neural network. The neural network includes an input layer, an intermediate layer, and an output layer, and each layer includes at least one neuron. Input data including image data acquired by the acquisition unit 21 is input to the input layer, and the input data is output to the intermediate layer. The intermediate layer extracts data of the operator 7 from the data input from the input layer. Then, the posture is detected based on the extracted data of the operator 7. The output layer outputs the data input from the intermediate layer as output data including posture data. As parameters in the intermediate layer, the positions of the joints of the operator 7 and the connection relationship between the joints are considered.

[0064] The calculation unit 23 has a function of identifying a reference position as a reference for calculating the positional deviation based on the image acquired by the acquisition unit 21, and using the identified reference position to calculate the positional deviation of the camera 11. The camera 11 of this embodiment is worn on at least one of the operator 7 and the item carried by the operator 7, but the worn camera 11 is not necessarily completely fixed, and there is a case where the wearing position of the camera 11 moves from the initial wearing position while the operator 7 performs a prescribed operation. For example, in Figure 2A In the case where the wearable terminal 3 is moved in a manner of rotating to the thumb side of the left hand 73c due to the rotation of the band 31 relative to the wrist, the acquisition unit 21 acquires Figure 5A The operator 7 rotates the image counterclockwise (in the direction of arrow 8). Figure 2C In the case where the camera 6 rotates clockwise due to the loosening of the belt 61, the acquisition unit 21 acquires Figure 5BIn such cases, since the posture of the operator 7 may not be correctly detected by the detection unit 22, the position deviation of the wearing position of the camera 11 is calculated by the function of the calculation unit 23, and the posture detected by the detection unit 22 is corrected by the function of the correction unit 24 described later. In this way, the posture of the operator 7 is correctly detected.

[0065] The position deviation of the camera 11 is calculated relative to a reference position of a prescribed state. The prescribed state is, for example, the initial state when the camera 11 is worn (hereinafter also referred to as the "initial state"), the state when the operator 7 starts a prescribed operation, the state before a prescribed time from a certain time point (such as the present), or the state at a certain time. How much the reference position has moved (how much deviation has occurred) relative to the reference position in these states is calculated, and the result of the calculation is used as the position deviation of the camera 11. When calculating the position deviation, the calculation unit 23 identifies a part of the body of the operator 7 contained in the image as the reference position, and uses the identified part of the body to calculate the position deviation.

[0066] The above-mentioned part of the body is a part of the body that is recognized in order to calculate the position deviation as simply as possible, and is a part that does not have a movable part such as a joint. Specifically, when the camera 11 is worn on the operator 7, it is a part that does not have a joint with the part on which the camera 11 is worn, and when the camera 11 is worn on the item carried by the operator 7, it is a part that does not have a joint with the part that contacts the item carried by the operator 7. The reason why the part itself does not have a part that can move through a joint is that when the position deviation is calculated using the part of the body as a reference position, it is necessary to distinguish whether the part at the reference position has moved due to the movable joint or the part at the reference position has moved due to the position deviation of the wearing position of the camera 11, and it is difficult to accurately calculate the position deviation. In addition, the reason why there is no joint between the part on which the camera 11 is worn and the part that contacts the item carried by the operator 7 is that if there is a joint between the camera 11 and the recognized part, it is difficult to distinguish whether the movement of the part set as the reference position is caused by the movable joint or by the position deviation of the wearing position.

[0067] However, since the position deviation of the portion where the imaging device 11 is worn cannot be calculated using the position where the imaging device 11 is worn, the portion where the imaging device 11 is worn and the portion in contact with the item worn by the operator 7 are excluded from the candidates for the portion to be the reference position. Here, the portion where the imaging device 11 is worn and the portion in contact with the item worn by the operator 7 are part of the portion of the body of the operator 7. That is, the portion identified as the portion to be the reference position and the portion where the imaging device 11 is worn may be the same portion. In addition, the portion identified as the portion to be the reference position and the portion in contact with the item worn by the operator 7 may be the same portion.

[0068] When detecting a part to be a reference position, a predetermined part of the body of the operator 7 included in the image data is detected by, for example, pattern matching. Then, when the predetermined part is recognized, the part is set as the reference position for calculating the position deviation. As an example, Figure 2A As shown in FIG. 1 , when the wearable terminal 3 is worn on the wrist of the left hand 73c, the forearm 73b of the left arm between the joints from the wrist to the left elbow is set in advance as the reference position. The forearm 73b of the left arm does not have a movable part through a joint, and there is no joint between the worn wearable terminal 3 and the forearm 73b of the left arm. In contrast, the left hand 73c has finger joints, so it will not be the reference position for calculating the position deviation. As another example, Figure 2B As shown in FIG. 1 , when the helmet 5 is worn on the head 71, the nose that is not covered by the helmet 5 is set in advance as the reference position. The nose itself does not have a part that can move through a joint, and there is no joint between the part covered by the helmet 5 and the nose. In contrast, the jaw and mouth are not the reference position for calculating the position deviation because the lower jaw moves through the jaw joint. And, as another example, Figure 2C As shown in FIG. 1 , when the camera 6 is worn on the part of the forearm 74b of the right arm near the elbow, the forearm 74b of the right arm is set in advance as the part that becomes the reference position. The forearm 74b of the right arm itself does not have a part that can move through a joint, and there is no joint between the worn camera 6 and the forearm 74b of the right arm (particularly, the wrist side of the forearm 74b of the right arm). In contrast, the right hand 74c does not become the reference position for calculating the position deviation because it moves through the joint of the right wrist.

[0069] As an example, in Figure 6A to Figure 6B , a method of calculating the positional deviation is shown in the case where a nose is recognized as a reference position for calculating the positional deviation and the recognized nose is set as the reference position. Fig. 6A It is shown by Figure 2B The camera 6 shown is a top view of an example of an image acquired when the head 71 is photographed. Fig. 6A The head 71 and the nose 711 are included, and the nose 711 is set as a reference position for calculating the position deviation. Fig. 6A The position deviation in the case of is relative to the position when the camera device 11 is first worn, that is, Fig. 6A In this case, if the helmet 5 is rotated counterclockwise, then Figure 6B As shown in FIG. 1 , compared with the position of the camera 11 when the helmet 5 is first worn, it appears that the head 71 is rotated clockwise by an angle α relative to the center 712 of the head 71 when the operator 7 is viewed from above. However, if the helmet 5 is worn neatly on the operator 7, the position of the nose 711 in the image does not change even if the operator 7 rotates. In fact, the operator 7 himself does not rotate, but only the helmet 5 rotates. The calculation unit 23 obtains the following information: Figure 6B In the case of an image in which the nose serving as the reference position has moved relative to the initial state, i.e., the position at which the camera device 11 is first worn, the distance and angle (in degrees) at which the nose has moved relative to the position of the nose 711 (i.e., the position of the center 712) in the initial state are detected. Figure 6B Next, the position deviation caused by the rotation of the helmet 5 (camera 6) is calculated based on the detected distance and angle. Then, the information of the calculated position deviation is output to the correction unit 24. In addition, the wearing position of the camera device 11 or the position of the nose 711 in the initial state is also referred to as the starting position for calculating the position deviation.

[0070] In addition, the part that becomes the reference position can also be identified from the time series images acquired by the acquisition unit 21 without presetting the part that becomes the reference position. For example, in the time series images captured at a specified period, a part that has only moved a distance within a specified range (for example, less than 5 mm) is detected, and the part is identified as the reference position. The specified range is a range that can detect the position deviation of the wearing position of the camera device 11. For example, in the case where the camera device 11 is worn upward on the ankle of the right leg, a part whose movement amount is less than 5 mm in the image is detected in the time series images captured at a period of one minute. In this case, since the operator 7 is photographed from the ankle of the right leg upward, the calf of the right leg hardly moves relative to the ankle. Therefore, the calf of the right leg is identified as the reference position.

[0071] Furthermore, when the acquisition unit 21 acquires a time series of images including a part serving as a reference position, the calculation unit 23 calculates the positional deviation based on the movement amount of the part in the time series of images. Fig. 6A and Figure 6B The movement of the nose 711 shown is as follows: Figure 6B The time series images during the period of rotation are calculated to calculate the direction and amount of movement of the nose 711 after wearing the helmet 5, and the calculation result is set as the position deviation. In this case, the starting position for calculating the position deviation can also be set to the initial position of the part in the time series images.

[0072] When calculating the positional deviation from the image acquired by the acquisition unit 21, the calculation unit 23 may use a positional deviation estimation model that has been previously learned so as to estimate the positional deviation from the position of a part set as a reference position in the image. The positional deviation estimation model is stored in, for example, Figure 1 The position deviation estimation model is shown in the database 14. In the position deviation estimation model, the image containing the operator 7 is associated with the position deviation calculated in the image in a corresponding manner, and when a certain image data is input, the position deviation corresponding to the input image data is output. The position deviation estimation model is Figure 1 The second machine learning unit 231 shown here performs machine learning. Specifically, the positional deviation estimation model is trained using past images and positional deviation calculation results stored in the database 14 so as to estimate appropriate positional deviation for input image data and output the result.

[0073] In addition, the position deviation estimation model may be a learned model obtained by machine learning. The learned model enables the computer (particularly the CPU 131 as a processor) to function in such a manner that when the image data acquired by the acquisition unit 21 is input as input data, the computer outputs output data including the position deviation based on the input data. By using such a learned model, the position deviation can be calculated even for the operator 7 who performs an operation other than the learned operation.

[0074] The learning model of the present embodiment is not particularly limited, but is, for example, a neural network. The neural network includes an input layer, an intermediate layer, and an output layer, and each layer includes at least one neuron. Input data including image data acquired by the acquisition unit 21 is input to the input layer, and the input data is output to the intermediate layer. The intermediate layer extracts data of the operator 7 from the data input from the input layer. Then, the position deviation is estimated based on the extracted data of the operator 7. The output layer outputs the data input from the intermediate layer as output data including data of the position deviation. As parameters in the intermediate layer, the positions of the joints of the operator 7 and the connection relationship between the joints are considered.

[0075] In addition, when the acquisition unit 21 acquires an image including a pattern for determining the position of the camera 11, the calculation unit 23 may recognize the pattern as a reference position and calculate the position deviation using the position of the camera 11 determined based on the pattern. The pattern is not particularly limited as long as it is a pattern that serves as a reference when calculating the position deviation in the image, and examples thereof include a plurality of crossed lines, a star-shaped mark, and the like. For example, a grid line pattern is given to a mask worn by a vehicle occupant, and the position deviation of the wearing position of the camera 11 is calculated using the grid lines. Furthermore, when the position of the camera 11 and the posture of the operator 7 are detected by the detection unit 22 using a detection device set around the operator 7, the calculation unit 23 may calculate the position deviation based on the position of the camera 11 and the posture of the operator 7 detected by the detection unit 22. For example, a camera set in front of an operator in an assembly plant is used to detect the position of the camera 11 and the posture of the operator 7, and the action of the operator 7 is estimated using the detected result.

[0076] The correction unit 24 has a function of correcting the posture detected by the detection unit 22 using the position deviation of the camera device 11 calculated by the calculation unit 23. The correction unit 24 uses the information of the position deviation calculated by the calculation unit 23 to geometrically transform the posture of the operator 7 geometrically expressed and acquired by the detection unit 22. For example, in the case of the helmet 5 Figure 6B When the helmet 5 is rotated counterclockwise by an angle α as shown, the detected posture is transformed by rotating it clockwise by an angle α. In this way, the posture of the operator 7 before the wearing position is deviated due to the rotation of the helmet 5, that is, the accurate posture of the operator 7 can be detected. In addition, when the helmet 5 deviates from the operator 7 in the front-to-back direction and the left-to-right direction, the detected posture is linearly transformed by at least one of translation, rotation, enlargement and reduction, so as to correct the movement of the helmet 5 (that is, the position deviation). Figure 2A The wearable terminal 3 and Figure 2C The camera 6 shown can also be corrected for positional deviation by performing the same geometric transformation.

[0077] The estimation unit 25 has a function of estimating the action of the operator 7 based on the posture of the operator 7 corrected by the correction unit 24. The action of the operator 7 refers to any action required to complete a predetermined operation, and includes an action of a vehicle occupant reaching for a switch of an onboard device to operate an onboard device such as an air conditioner, an action of an assembly plant worker taking out a torque wrench from a tool box, and an action of a dealer maintenance worker touching a switch for raising and lowering a lift for lifting a vehicle. Examples of the action include an action of a vehicle occupant pressing a switch for raising and lowering a vehicle window glass, an action of a vehicle occupant touching a touch panel to change a map display of a navigation device, an action of an assembly plant worker connecting a coupler connected to a sensor with a coupler connected to an electronic control unit (ECU), an action of an assembly plant worker using a tool to tighten bolts to mount an exhaust manifold to an engine module, an action of a dealer maintenance worker mounting a spark plug to an engine, and an action of a dealer maintenance worker tightening bolts using a torque wrench.

[0078] For example, the estimation unit 25 Figure 4A When the posture shown is detected, according to Figure 4A The positions of the points P1 to P3 and the shape of the straight line drawn from the points P1 to P3 are estimated when the operator 7, who is a passenger of the vehicle, reaches out his right hand (not shown) to the operating part of the vehicle-mounted device when he wants to operate the vehicle-mounted device. Figure 4B When the posture shown is detected, according to Figure 4B The positions of points P7 to P9 and points P10 to P12 and the shapes of the straight lines drawn between points P7 to P9 and points P10 to P12 are estimated to be when the operator 7, who is a worker in the assembly factory, wants to grasp the tool with his left hand 73c and right hand 74c when he wants to operate the tool for assembly. Figure 4C When the posture shown is detected, according to Figure 4C The positions of the points P19 to P32 shown and the shape of the straight line drawn from the points P19 to P32 are estimated to be the operator 7, a factory worker, extending his fingers to grab the tool for assembly with his right hand 74c. For example, pattern matching is used to estimate such an action. Specifically, characteristic parts such as the positions of the joints of the operator 7, the relationship between the joints, and the shape of the body parts connecting the joints are extracted from the input posture data, and the extracted characteristic parts are compared with the class dictionary stored in the database 14, thereby determining which class the characteristic parts belong to. In the class dictionary, if the action corresponding to the posture is set as a class, the action can be estimated based on the posture.

[0079] When estimating the behavior, the estimation unit 25 can use a behavior estimation model that is learned in advance so as to estimate the behavior of the operator 7 based on the posture of the operator 7 detected by the detection unit 22 and corrected by the correction unit 24. The behavior estimation model is stored in, for example, Figure 1 The action estimation model is a database 14 shown. In the action estimation model, the detected posture of the operator 7 is associated with the action of the operator 7 in a corresponding manner. When the data of a certain posture is input, the action of the operator 7 corresponding to the input posture data is output. The action estimation model is Figure 1 The third machine learning unit 251 shown in the figure performs machine learning. Specifically, the behavior estimation model uses the past detection results of the posture of the operator 7 stored in the database 14 and the estimated behavior of the operator 7 to learn in a manner that estimates appropriate behavior for the input posture data and outputs the result.

[0080] In addition, the action estimation model may be a learned model obtained by machine learning. The learned model enables the computer (particularly the CPU 131 as a processor) to function in a manner that, when the data of the posture detected by the detection unit 22 is input as input data, outputs output data including the action of the operator 7 based on the input data. By using such a learned model, the action can be appropriately detected even for the operator 7 who performs an operation other than the learned operation. In addition, the input posture data may be corrected by the correction unit 24 or not.

[0081] The learned model of the present embodiment is not particularly limited, and may be, for example, a neural network. The neural network includes an input layer, an intermediate layer, and an output layer, and each layer includes at least one neuron. Input data including data of a posture detected by the detection unit 22 is input to the input layer, and the input data is output to the intermediate layer. The intermediate layer extracts data of the posture of the operator 7 from the data input from the input layer. Next, the action is estimated based on the extracted data of the posture of the operator 7. The output layer outputs the data input from the intermediate layer as output data including data of the action. As parameters in the intermediate layer, for example, the positions of points representing the joints of the operator 7 and the shapes of straight lines representing the connection relationship between the joints are considered.

[0082] In addition, there are cases where a plurality of actions are estimated for the posture of the operator 7, but since the operator 7 in the present embodiment performs a predetermined operation, if the attribute of the operator 7 is determined, the estimation unit 25 can determine the action corresponding to the posture. The attribute of the operator 7 refers to, for example, whether the operator 7 is a passenger of the vehicle, a worker working on an assembly line in an assembly plant, or a maintenance worker in a maintenance plant of a vehicle dealer, and features of the position of the operator 7 and the place where the operator 7 is present.

[0083] The output unit 26 has a function of outputting the action estimated by the estimation unit 25 to the display device 12. The action of the operator 7 output from the output unit 26 is received by the display device 12. And, it is displayed on the display device 12 and presented to the operator 7 and the supervisor of the operator 7. The operator 7 and the supervisor can confirm whether the action of the operator 7 is the action required for the prescribed operation, whether the operator 7 performs the prescribed operation in the prescribed process, whether the operator 7 skips the necessary process, etc. by confirming the action presented by the display device 12. In addition, it is also possible to replace the operator 7 and the supervisor, and determine whether the action of the operator 7 is the action required for the prescribed operation, whether the operator 7 performs the prescribed operation in the prescribed process, whether the operator 7 skips the necessary process, etc. in the processing unit 2. In this case, the prescribed operation, the process of the prescribed operation, the action required for the prescribed operation, etc. are stored in the database 14 and are obtained by the processing unit 2 as needed.

[0084] [Processing in image processing system]

[0085] Reference Figure 7 The process of processing information by the image processing device 13 will be described. Figure 7 This is an example of a flowchart showing information processing in the image processing system 1 of the present embodiment. While the operator 7 performs a predetermined operation, the CPU 131 as a processor of the image processing device 13 executes the processing described below at predetermined time intervals.

[0086] First, in step S1, the image data captured by the camera 11 is acquired by the function of the acquisition unit 21. In the subsequent step S2, it is determined whether the acquired image data includes the operator 7. If the acquired image data does not include the operator 7, the process returns to step S1. On the other hand, if the acquired image data includes the operator 7, the process proceeds to step S3.

[0087] In step S3, the posture of the operator 7 included in the image is detected by the function of the detection unit 22. In the next step S4, it is determined whether the posture of the operator 7 is properly detected. If the posture of the operator 7 is not properly detected, the process returns to step S1. On the other hand, if the posture of the operator 7 is properly detected, the process proceeds to step S5.

[0088] In step S5, the function of the calculation unit 23 calculates the degree of deviation of the wearing position of the camera 11 from the reference position of the initial state, for example. In the next step S6, it is determined whether the wearing position of the camera 11 has deviated. Specifically, it is determined whether the calculated amount of position deviation falls within the range in which the posture of the operator 7 can be correctly detected. If the calculated amount of position deviation falls within the range in which the posture of the operator 7 can be correctly detected, the process proceeds to step S8. On the other hand, if the calculated amount of position deviation exceeds the range in which the posture of the operator 7 can be correctly detected, the process proceeds to step S7.

[0089] In step S7, the posture of the operator 7 is corrected based on the deviation calculated by the calculation unit 23 by the function of the correction unit 24. In the following step S8, the action of the operator 7 is estimated from the posture of the operator 7 by the function of the estimation unit 25. In the following step S9, it is determined whether the action of the operator 7 is properly estimated. If the action of the operator 7 is not properly estimated, the execution of the routine is stopped to end the processing of the information. On the other hand, if the action of the operator 7 is properly estimated, the process proceeds to step S10.

[0090] In step S10, the data including the estimated action is output from the image processing device 13 to the display device 12 by the function of the output unit 26. In the following step S11, the function of the processing unit 2 determines whether the operator 7 has properly performed the prescribed operation based on the action estimated by the estimation unit 25. At this time, the result of the determination may also be output to the display device 12. When the processing in step S11 is completed, the execution of the routine is stopped to end the processing of the information. In addition, steps S2, S4, S6, and S9 to S11 are not essential steps and can be provided as needed.

[0091] [Embodiments of the present invention]

[0092] As described above, according to the present embodiment, there is provided an image processing device 13, which includes: an acquisition unit 21 that acquires an image including the operator 7 from the imaging device 11 worn on at least one of the operator 7 who performs a predetermined operation and an item carried by the operator 7; a detection unit 22 that detects the posture of the operator 7 from the image; a calculation unit 23 that calculates the position deviation of the imaging device 11 based on the image using a reference position as a reference for calculating the position deviation; and a correction unit 24 that corrects the posture using the position deviation. Thus, even if the wearing position of the imaging device 11 (e.g., a camera) is deviated, the posture of the operator 7 can be correctly identified.

[0093] In addition, the image processing device 13 according to the present embodiment further includes an estimation unit 25, which estimates the behavior of the operator 7 from the corrected posture using a behavior estimation model learned in advance in a manner to estimate the behavior of the operator 7 from the posture. Thus, the behavior of the operator 7 can be estimated by associating the posture of the operator 7 with the behavior.

[0094] In addition, according to the image processing device 13 of this embodiment, the detection unit 22 detects the posture of the operator 7 from the image using a posture estimation model that has been learned in advance to estimate the posture of the operator 7 from the image. This makes it possible to perform high-precision posture detection using previous detection results.

[0095] In addition, according to the image processing device 13 of this embodiment, the calculation unit 23 recognizes a part of the body of the operator 7 included in the image as the reference position, and calculates the position deviation relative to the reference position in a predetermined state. Thus, it is possible to realize simple and accurate calculation of the position deviation using a part of the body.

[0096] Furthermore, according to the image processing device 13 of this embodiment, when the imaging device 11 is worn by the operator, the reference position is a part of the body of the operator 7 other than the part where the imaging device 11 is worn, which has no part that can move through a joint and has no joint with the part where the imaging device 11 is worn; and when the imaging device 11 is worn on the article, the reference position is a part of the body of the operator 7 other than the part in contact with the article, which has no part that can move through a joint and has no joint with the part in contact with the article carried by the operator 7. Thus, the body part can be specified more accurately.

[0097] In addition, according to the image processing device 13 of this embodiment, the acquisition unit 21 acquires time-series images including the reference position, and the calculation unit 23 calculates the position deviation using the movement amount of the reference position in the time-series images. Thus, the position deviation of the imaging device 11 can be calculated more accurately.

[0098] In the image processing device 13 of this embodiment, the calculation unit 23 calculates the position deviation using a position deviation estimation model learned in advance to estimate the position deviation from the position of the reference position in the image. Thus, the amount of position deviation can be calculated only by inputting an image.

[0099] In addition, according to the image processing device 13 of this embodiment, the acquisition unit 21 acquires an image including a pattern for determining the position of the imaging device 11, the calculation unit 23 recognizes the pattern as the reference position, and calculates the position deviation using the position of the imaging device 11 determined based on the pattern. Thus, the position deviation of the imaging device 11 can be calculated more accurately.

[0100] In addition, according to the image processing device 13 of this embodiment, the detection unit 22 detects the position of the imaging device 11 and the posture of the operator 7 using the detection device provided around the operator 7, and the calculation unit 23 calculates the position deviation based on the position of the imaging device 11 and the posture of the operator 7 detected by the detection unit 22. Thus, the position deviation of the imaging device 11 can be calculated more accurately.

[0101] In addition, according to the present embodiment, there is provided an image processing method, which is an image processing method executed by a processor, in which the processor performs the following processing: acquiring an image including the operator 7 from a camera 11 worn on at least one of the operator 7 who performs a predetermined operation and an item carried by the operator 7; detecting the posture of the operator 7 from the image; calculating the position deviation of the camera 11 based on the image using a reference position as a reference for calculating the position deviation; and correcting the posture using the position deviation. Thus, even if the wearing position of the camera 11 (e.g., a camera) is deviated, the posture of the operator 7 can be correctly identified.

[0102] Description of Reference Numerals

[0103] 1: Image processing system; 11: Camera device; 12: Display device; 13: Image processing device; 131: CPU; 132: ROM; 133: RAM; 14: Database; 2: Processing unit; 21: Acquisition unit; 22: Detection unit; 221: First machine learning unit; 23: Calculation unit; 231: Second machine learning unit; 24: Correction unit; 25: Estimation unit; 251: Third machine learning unit; 26: Output unit; 3: Wearable terminal; 31: Band; 4, 4a, 4b: field of view; 5: helmet; 51: brim; 52: bracket; 6: camera; 61: belt; 7: operator (person); 71: head; 711: nose; 712: center; 72: neck; 73a: upper arm of left arm; 73b: forearm of left arm; 73c: left hand; 74a: upper arm of right arm; 74b: forearm of right arm; 74c: right hand; 75: torso; 76: left leg; 77: right leg; 8, 9: arrows; P1~P32: points; α: angle.

Claims

1. An image processing device comprising: An acquisition unit that acquires an image including the person from an imaging device worn on at least one of a person and an article carried by the person; a detection unit configured to detect a posture of the person from the image; a calculation unit that calculates a positional deviation of the imaging device based on the image using a reference position as a reference for calculating the positional deviation; as well as a correction unit that corrects the posture using the position deviation, wherein the calculation unit recognizes a part of the body of the person included in the image as the reference position to calculate the position deviation from the reference position in a predetermined state, When the camera device is worn on the person, the reference position is a part of the body of the person other than the part on which the camera device is worn, which part itself has no part movable by a joint and has no joint with the part on which the camera device is worn, When the camera device is worn on the article, the reference position is a part of the person's body other than the part in contact with the article, where there is no part that can move through a joint and there is no joint between the part in contact with the article carried by the person.

2. The image processing device according to claim 1, wherein: The device further includes an estimating unit that estimates the behavior of the person from the corrected posture using a behavior estimation model that has been previously learned so as to estimate the behavior of the person from the posture.

3. The image processing device according to claim 1 or 2, wherein: The detection unit detects the posture of the person from the image using a posture estimation model that is learned in advance so as to estimate the posture of the person from the image.

4. The image processing device according to claim 1, wherein: The acquisition unit acquires a time-series image including the reference position. The calculation unit calculates the positional deviation using the amount of movement of the reference position in the time-series images.

5. The image processing device according to claim 1, wherein: The calculation unit calculates the positional deviation using a positional deviation estimation model that is learned in advance so as to estimate the positional deviation from the position of the reference position in the image.

6. The image processing device according to claim 1 or 2, wherein: The detection unit detects the position of the camera device and the posture of the person using a detection device provided around the person, The calculation unit calculates the positional deviation based on the position of the imaging device and the posture of the person detected by the detection unit.

7. An image processing method is an image processing method executed by a processor, in which: The processor performs the following processing: acquiring an image including the person from a camera device worn on at least one of a person and an item carried by the person; detecting a posture of the person from the image; calculating a positional deviation of the imaging device using a reference position as a reference for calculating the positional deviation based on the image; and using the position deviation to correct the posture, in, The processor also performs the following processing: recognizing a part of the body of the person included in the image as the reference position to calculate the position deviation from the reference position in a predetermined state, When the camera device is worn on the person, the reference position is a part of the body of the person other than the part on which the camera device is worn, which part itself has no part movable by a joint and has no joint with the part on which the camera device is worn, When the camera device is worn on the article, the reference position is a part of the person's body other than the part in contact with the article, where there is no part that can move through a joint and there is no joint between the part in contact with the article carried by the person.

Images