Processing device, processing method, and program

Abstract

This processing device, if a human body of a prescribed height is captured in a prescribed direction in an image: determines whether the human body is facing forward, backwards, left or right, and determines the degree to which the human body is facing sideways, on the basis of a reference value (theoretical value) indicating an in-image lateral distance between a first key point and a second key point of the human body, the in-image lateral distance (actual measurement value) between the first key point and the second key point, and the positional relationship between a key point on the right side of the human body and a key point on the left side of the human body on the image; and determines the direction of the human body in the image on the basis of the determination results.

Description

Processing device, processing method, and recording medium

[0001] The present invention relates to a processing device, a processing method, and a program.

[0002] Techniques related to the present invention are disclosed in Patent Documents 1 to 4.

[0003] The technology disclosed in Patent Document 1 derives a customer's movement path within a store, including the orientation of the customer's body, based on posture information. The posture information includes information on the orientation of the customer's face. The orientation of the customer's face is determined based on the relative positions of each feature by extracting the customer's outline, hair area, facial area, etc. using image processing.

[0004] The technology disclosed in Patent Document 2 generates posture estimation data indicating the positions of a person's joint points. Next, the technology calculates the length of a predetermined part of the human body (such as the upper arm) based on the posture estimation data, and calculates the length of the human body's torso by multiplying the length of the predetermined part by a predetermined value. The technology also calculates the length of the human body's shoulder width based on the posture estimation data. The technology then determines whether the human body is oriented sideways based on the ratio between the length of the human body's torso and the length of the shoulder width. Specifically, the technology determines that the body is oriented sideways when the shoulder width is shorter than one-third of the length of the torso.

[0005] The technology disclosed in Patent Document 3 estimates a person's height based on a two-dimensional skeletal structure that indicates the positions of multiple key points on the person.

[0006] The technology disclosed in Patent Document 4 searches for images containing human bodies with similar postures or movements based on a two-dimensional skeletal structure that indicates the positions of multiple key points of a person.

[0007] Japanese Patent Publication No. 2019-139321 Japanese Patent Publication No. 2022-091612 Japanese Patent No. 7173341 International Publication No. 2021 / 084677

[0008] Patent Document 2 discloses a technology for determining the orientation of a human body based on data indicating the positions of multiple key points on a person. Specifically, the technology disclosed in Patent Document 2 determines whether a human body is oriented sideways based on data indicating the positions of multiple key points on a person. The technology disclosed in Patent Document 2 can determine whether a human body is oriented sideways. If the orientation of a human body can be determined more precisely, the versatility of the technology for determining the orientation of a human body based on data indicating the positions of multiple key points on a person will be expanded. None of Patent Documents 1 to 4 discloses this problem or a means for solving it.

[0009] In view of the above-mentioned problems, one example of the object of the present invention is to provide a processing device, a processing method, and a program that can more precisely determine the orientation of a human body in a technology for determining the orientation of a human body based on data indicating the positions of multiple key points on a person.

[0010] According to one aspect of the present invention, there is provided a processing device comprising: a skeletal structure detection means for detecting a plurality of key points of a human body included in an image; a reference information generation means for generating reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points; a reference value calculation means for calculating, based on the reference information, a reference value indicating the lateral distance on the image between a first key point and a second key point of the human body when the human body is shown in the image in a predetermined orientation; an actual measurement calculation means for calculating, based on the plurality of key points, an actual measurement value indicating the lateral distance on the image between the first key point and the second key point of the human body; and an orientation determination means for determining which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is, based on the reference value, the actual measurement value, and the positional relationship on the image between the key point on the right side of the human body and the key point on the left side of the human body.

[0011] According to one aspect of the present invention, there is provided a processing method in which one or more computers detect a plurality of key points of a human body included in an image, generate reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points, calculate a reference value indicating the lateral distance on the image between a first key point and a second key point of the human body when the human body is shown in the image in a predetermined orientation based on the reference information, calculate an actual measurement value indicating the lateral distance on the image between the first key point and the second key point of the human body based on the plurality of key points, and determine which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is based on the reference value, the actual measurement value, and a positional relationship on the image between the key point on the right side of the human body and the key point on the left side of the human body.

[0012] According to one aspect of the present invention, there is provided a program that causes a computer to function as: a skeletal structure detection means that detects a plurality of key points of a human body included in an image; a reference information generation means that generates reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points; a reference value calculation means that calculates, based on the reference information, a reference value that indicates the lateral distance on the image between a first key point and a second key point of the human body when the human body is shown in the image in a predetermined orientation; an actual measurement calculation means that calculates, based on the plurality of key points, an actual measurement value that indicates the lateral distance on the image between the first key point and the second key point of the human body; and an orientation determination means that determines which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is, based on the reference value, the actual measurement value, and the positional relationship on the image between the key point on the right side of the human body and the key point on the left side of the human body.

[0013] According to one aspect of the present invention, in a technology for determining the orientation of a human body based on data indicating the positions of multiple key points of a person, a processing device, processing method, and program are realized that determine the orientation of a human body more precisely.

[0014] The above and other objects, features and advantages will become more apparent from the following preferred embodiments and the accompanying drawings.

[0015] FIG. 1 is a diagram showing an example of a functional block diagram of a processing device. FIG. 2 is a diagram showing an example of a hardware configuration of a processing device. FIG. 3 is a diagram showing an example of key points of a human body to be detected. FIG. 4 is a diagram showing an example of key points of a detected human body. FIG. 5 is a diagram for explaining a process of determining the orientation of a human body on an image. FIG. 6 is another diagram for explaining a process of determining the orientation of a human body on an image. FIG. 7 is a flowchart showing an example of a processing flow of a processing device. FIG. 8 is another example of a functional block diagram of a processing device. FIG. 9 is a diagram for explaining a process of determining the orientation of a detection object on an image. FIG. 10 is a diagram showing another example of a functional block diagram of a processing device.

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0017] 1 is a functional block diagram showing an overview of a processing device 10 according to a first embodiment. The processing device 10 includes a skeletal structure detection unit 11, a reference information generation unit 12, a reference value calculation unit 13, an actual measurement value calculation unit 14, and an orientation determination unit 15.

[0018] The skeletal structure detection unit 11 detects a plurality of key points of the human body included in the image.

[0019] The reference information generating unit 12 generates reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points detected by the skeletal structure detecting unit 11 .

[0020] The reference value calculation unit 13 calculates, based on the reference information generated by the reference information generation unit 12, a reference value indicating the lateral distance on the image between the first key point and the second key point of the human body when the human body is shown in the image in a specified orientation.

[0021] The actual measurement value calculation unit 14 calculates an actual measurement value indicating the lateral distance on the image between the first key point and the second key point of the human body based on the multiple key points detected by the skeletal structure detection unit 11.

[0022] The orientation determination unit 15 determines which of multiple orientations represented by angles from 0° to 360° the orientation of the human body on the image is based on the reference value, the actual measurement value, and the positional relationship on the image between the key points on the right side of the human body and the key points on the left side of the human body.

[0023] According to the processing device 10 of the first embodiment, in a technique for determining the orientation of a human body based on data indicating the positions of multiple key points of a person, the orientation of the human body can be determined more precisely.

[0024] Second Embodiment Overview A processing apparatus 10 according to a second embodiment is a specific embodiment of the processing apparatus 10 according to the first embodiment. The configuration of the processing apparatus 10 will be described in detail below.

[0025] "Hardware Configuration" An example of the hardware configuration of the processing device 10 will be described. Each functional unit of the processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. Software includes programs that are pre-loaded in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0026] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a processing device 10. As shown in FIG. 2, the processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0027] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, touch panel, etc. Examples of output devices include a display, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0028] "Functional Configuration" Next, the functional configuration of the processing device 10 of this embodiment will be described in detail. Fig. 1 shows an example of a functional block diagram of the processing device 10 of this embodiment. As shown in the figure, the processing device 10 of this embodiment has a skeletal structure detection unit 11, a reference information generation unit 12, a reference value calculation unit 13, an actual measurement value calculation unit 14, and an orientation determination unit 15.

[0029] The skeletal structure detection unit 11 detects a plurality of key points of the human body included in the image.

[0030] The concept of "image" includes both still images and moving images. Moving images are made up of a plurality of time-series frame images.

[0031] The skeletal structure detection unit 11 detects N (N is an integer equal to or greater than 2) key points of the human body contained in an image. When a moving image is to be processed, the skeletal structure detection unit 11 performs processing to detect key points for each frame image. The skeletal structure detection unit 11 detects N key points using a skeletal estimation technology such as OpenPose. The human skeletal structure detected by this technology consists of "key points," which are characteristic points such as joints, and "bones (bone links)," which indicate the links between key points.

[0032] Fig. 3 shows an example of a human skeletal structure detected by the skeletal structure detection unit 11. The skeletal structure detection unit 11 detects a human skeletal structure such as that shown in Fig. 3 from a two-dimensional image using a skeletal estimation technique such as OpenPose.

[0033] The skeletal structure detection unit 11, for example, extracts feature points that can be key points from an image and detects N key points of the human body by referring to information obtained by machine learning of the image of the key points. The N key points to be detected are determined in advance. The number of key points to be detected (i.e., the number N) and which parts of the human body are to be detected as key points vary, and any variation can be adopted.

[0034] In this embodiment, it is assumed that 18 (N=18) key points are defined as detection targets, as shown in Fig. 3. The 18 key points include a nose C0, a left eye C1, a right eye C2, a left ear C3, a right ear C4, a left shoulder C5, a right shoulder C6, a left elbow C7, a right elbow C8, a left hand C9, a right hand C10, a left waist C11, a right waist C12, a left knee C13, a right knee C14, a left foot C15, a right foot C16, and a neck C17.

[0035] Returning to FIG. 1 , the reference information generating unit 12 generates reference information based on the plurality of key points detected by the skeletal structure detecting unit 11 .

[0036] The "reference information" indicates the height of the human body shown in the image or the length of a predetermined part of the human body.

[0037] The "predetermined part of the human body" is a part whose length has a strong correlation with the height of the human body. Examples of such a predetermined part of the human body include, but are not limited to, the trunk, legs, and arms.

[0038] The reference information indicates the height of a human body or the length of a predetermined part of the human body as a length on the image. For example, the height of a human body and the length of a predetermined part of the human body are indicated by the number of pixels or the distance in a coordinate system set on the image (the distance between two points on the coordinate system). Such reference information can be calculated using any technique. For example, the techniques disclosed in Patent Document 2 and Patent Document 3 may be used, but are not limited to these.

[0039] The reference value calculation unit 13 calculates a reference value based on the reference information generated by the reference information generation unit 12 .

[0040] The "reference value" indicates the horizontal distance on the image between the first key point and the second key point of a human body when the human body has the height or the length of a specified part indicated in the reference information and is shown in the image in a specified orientation. The distance is indicated as a length on the image, such as the number of pixels or the distance in a coordinate system set on the image (the horizontal distance between two points in the coordinate system).

[0041] The "predetermined orientation" is a design factor. In this embodiment, the predetermined orientation is the orientation facing the camera.

[0042] The "first keypoint and second keypoint" are two of the N keypoints to be detected. The two keypoints that are paired and located on either side of the center of the human body are used as the first keypoint and the second keypoint.

[0043] The first key point and the second key point are, for example, any of the following examples: (1) left eye C1 and right eye C2, (2) left ear C3 and right ear C4, (3) left shoulder C5 and right shoulder C6, (4) left elbow C7 and right elbow C8, (5) left hand C9 and right hand C10, (6) left hip C11 and right hip C12, (7) left knee C13 and right knee C14, and (8) left foot C15 and right foot C16.

[0044] Among the above examples, it is preferable to use any of (1), (2), (3), and (6) as the first and second keypoints, which have a relatively small degree of freedom in the distance between the first and second keypoints. As will be described below, in this embodiment, the degree to which the human body is oriented sideways on the image is determined based on the lateral distance on the image between the first and second keypoints on the human body. For this reason, it is not preferable to use keypoints whose distance may vary due to factors other than the orientation of the human body.

[0045] Here, we will explain the process of calculating a reference value that indicates the lateral distance on an image between a first key point and a second key point of a human body when the human body is shown in an image in a specified orientation, based on the height of the human body or the length of a specified part of the human body indicated in the reference information.

[0046] A calculation model for calculating a reference value from reference information is generated in advance and stored in the processing device 10. The reference value calculation unit 13 calculates the reference value based on the calculation model.

[0047] The calculation model may have various configurations. A simple calculation model may calculate a reference value by multiplying the height of a human body or the length of a predetermined part of the human body indicated in the reference information by a predetermined correction coefficient. The correction coefficient is determined in advance based on the relationship (ratio) between the "height of a human body or the length of a predetermined part of the human body" and the "lateral distance between the first key point and the second key point" when the human body is captured in an image in a predetermined orientation (e.g., facing the camera). While this relationship may differ for each human body, the difference is small and can be generalized. For example, multiple human bodies may be surveyed, and the ratio may be calculated for each human body. Then, the correction coefficient may be a statistical value (average, mode, median) of the ratios for multiple human bodies. Note that the calculation model illustrated here is merely an example and is not limited to this.

[0048] The actual measurement value calculation unit 14 calculates an actual measurement value indicating the lateral distance on the image between the first key point and the second key point of the human body based on the multiple key points detected by the skeletal structure detection unit 11.

[0049] The "measured value" is expressed as a length on the image, such as the number of pixels or a distance in a coordinate system set on the image (the distance between two points in the coordinate system).

[0050] FIG. 4 shows an example of the calculated horizontal distance W. In the example shown, key points C5 and C6 are the first and second key points. The "horizontal distance" is the distance between the first and second key points in the horizontal direction of the image, and does not include the vertical distance component. For example, in a coordinate system in which the horizontal direction of the image is the x-axis direction and the vertical direction of the image is the y-axis direction, if the coordinates of the first key point are (x １ , y １ ), the coordinates of the second keypoint are (x ２ , y ２ ), then (x １ -x ２ ) is the horizontal distance.

[0051] Returning to FIG. 1, the orientation determination unit 15 determines which of a plurality of orientations the human body is facing on the image.

[0052] The "orientation of the human body" refers to the orientation of a specific part of the body. For example, the orientation of the face or the orientation of the torso (or the orientation of the stomach) is defined as the orientation of the human body.

[0053] The "orientation of the human body on the image" is represented by an angle P, as shown in Fig. 5. The first orientation is defined as 0°, and the angle P takes values ​​between 0° and 360°. Fig. 5 shows a human body H (a human body included in the image) to be photographed and a camera photographing the human body H, as viewed from above.

[0054] In the illustrated example, the state in which the human body H faces the camera directly is defined as P=0°. Then, the state in which the human body H is rotated 90° to the left from the camera's perspective, and faces straight to the side (left from the camera's perspective), is defined as P=90°. Then, the state in which the human body H is rotated another 90° in the same direction from that state, and faces its back to the camera, is defined as P=180°. Then, the state in which the human body H is rotated another 90° in the same direction from that state, and faces straight to the side (right from the camera's perspective), is defined as P=270°.

[0055] The orientation determination unit 15 determines which of the multiple orientations represented by the angle P corresponds to the orientation of the human body H on the image. 360 types of orientations may be defined in increments of 1°. １ 360 / n in increments of ° (n is greater than 1) １ A variety of orientations may be defined. ２ 360 / n in increments of ° (n is greater than 0 and less than 1) ２ A variety of orientations may be defined.

[0056] 6 shows an example of the definition of multiple orientations. In the illustrated example, eight orientations are defined in increments of 45°. A representative value of the angle P is associated with each orientation. That is, eight orientations are defined, with P=0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° as representative values. In this example, the orientation determination unit 15 determines which of the eight orientations the human body H in the image is facing.

[0057] The multiple orientations are distinguished from one another based on whether the human body is facing forward or backward, whether the human body is facing right or left, and the degree to which the human body is facing sideways. That is, the orientation determination unit 15 determines whether the human body included in the image is facing forward or backward, whether the human body is facing right or left, and the degree to which the human body is facing sideways. Then, based on the result of this determination, the orientation determination unit 15 determines which of the multiple orientations described above the orientation of the human body in the image is.

[0058] The orientation determination unit 15 makes this determination based on the "reference value calculated by the reference value calculation unit 13," the "actual measurement value calculated by the actual measurement value calculation unit 14," and the "positional relationship on the image between the key points on the right side of the human body and the key points on the left side of the human body." This will be described in detail below.

[0059] 6, "facing forward" is a state in which the human body H faces toward the camera. "facing backward" is a state in which the human body H faces away from the camera.

[0060] The orientation determination unit 15 determines whether the human body is facing forward or backward on the image based on the relative positional relationship in the left-right direction on the image between the paired key points on the right and left parts of the human body.

[0061] The "paired key points on the right side and left side of the human body" are, for example, any of the following examples: (1) left eye C1 and right eye C2, (2) left ear C3 and right ear C4, (3) left shoulder C5 and right shoulder C6, (4) left elbow C7 and right elbow C8, (5) left hand C9 and right hand C10, (6) left waist C11 and right waist C12, (7) left knee C13 and right knee C14, and (8) left foot C15 and right foot C16.

[0062] It should be noted that the key point (1) is difficult to detect when the person is photographed with their back to the camera, so it is preferable to use one of the key points (2) to (8).

[0063] The "relative positional relationship in the left-right direction on the image" refers to which is located on the right (or left) side on the image.

[0064] 4, when the human body is facing forward, the key points on the right side (such as the right shoulder C6) are located to the left of the key points on the left side (such as the left shoulder C5) in the image. On the other hand, although not shown, when the human body is facing backward, the key points on the right side (such as the right shoulder C6) are located to the right of the key points on the left side (such as the left shoulder C5) in the image.

[0065] Based on this relationship, the orientation determination unit 15 determines whether the human body is facing forward or backward in the image. Specifically, the orientation determination unit 15 determines that the human body is facing forward when a key point on the right side (such as the right shoulder C6) is located to the left of a key point on the left side (such as the left shoulder C5) in the image. On the other hand, the orientation determination unit 15 determines that the human body is facing backward when a key point on the right side (such as the right shoulder C6) is located to the right of a key point on the left side (such as the left shoulder C5) in the image.

[0066] 6, "right facing" refers to a state in which the human body is facing to the right as seen from the camera, and "left facing" refers to a state in which the human body is facing to the left as seen from the camera.

[0067] The orientation determination unit 15 determines whether the human body is facing right or left in the image based on at least one of the following: - Whether or not key points on the right side of the human body and key points on the left side of the human body are detected in the detection of multiple key points; - The orientation of the elbow indicated by multiple key points; - The orientation of the knee indicated by multiple key points.

[0068] (Determination based on whether key points on the right side of the human body and key points on the left side of the human body are detected in detecting multiple key points) The "key points on the right side of the human body" are the right eye C2, right ear C4, right shoulder C6, right elbow C8, right hand C10, right hip C12, right knee C14, right foot C16, etc. The "key points on the left side of the human body" are the left eye C1, left ear C3, left shoulder C5, left elbow C7, left hand C9, left hip C11, left knee C13, left foot C15, etc.

[0069] When a photograph of a human body is taken facing right (facing "back-right" or "forward-right" in FIG. 6 ), the left side of the human body is likely to be hidden by other parts of the body in the image. As a result, key points on the left side of the body are difficult to detect in the detection process by the skeletal structure detection unit 11. On the other hand, when a photograph of a human body is taken facing left (facing "back-left" or "forward-left" in FIG. 6 ), the right side of the human body is likely to be hidden by other parts of the body in the image. As a result, key points on the right side of the human body are difficult to detect in the detection process by the skeletal structure detection unit 11.

[0070] Based on this relationship, the orientation determination unit 15 can determine whether the human body is facing right or left on the image.

[0071] For example, if any key point on the right side of the human body is undetected, the orientation determination unit 15 can determine that the human body is facing left. If any key point on the left side of the human body is undetected, the orientation determination unit 15 can determine that the human body is facing right.

[0072] As another example, the orientation determination unit 15 may count the number of undetected key points in each of the key points on the right side of the human body and the key points on the left side of the human body. Then, if the number of undetected key points is greater on the right side of the human body, the orientation determination unit 15 may determine that the human body is facing left. Also, if the number of undetected key points is greater on the left side of the human body, the orientation determination unit 15 may determine that the human body is facing right.

[0073] (Determination based on elbow orientation indicated by multiple key points) The "elbow orientation" is the direction in which the convex surface faces when the elbow is bent. Due to the skeletal structure of the human body, the orientation of the elbow tends to be opposite to the orientation of the human body (such as the orientation of the face). Based on this relationship, the orientation determination unit 15 can determine whether the human body is facing right or left in the image.

[0074] Specifically, when the elbow of the human body is facing right, the orientation determination unit 15 determines that the human body is facing left. When the elbow of the human body is facing left, the orientation determination unit 15 determines that the human body is facing right.

[0075] The direction of the elbow can be defined as the direction from an arbitrary point (e.g., midpoint) on the line segment connecting the key points of the shoulder and hand toward the elbow key point. Based on this definition, the direction determination unit 15 can determine the direction of the elbow of the human body on the image. In the example shown in FIG. 4, the direction of the right elbow is direction d2 from an arbitrary point on the line segment connecting the right shoulder C6 and the right hand C10 toward the right elbow C8. As can be seen from the figure, in this example, the direction of the right elbow is facing right. Therefore, the direction determination unit 15 determines that the human body is facing left.

[0076] (Determination based on knee orientation indicated by multiple key points) The "knee orientation" is the direction in which the convex surface faces when the knee is bent. Due to the skeletal structure of the human body, the orientation of the knee tends to be the same as the orientation of the human body (such as the orientation of the face). Based on this relationship, the orientation determination unit 15 can determine whether the human body is facing right or left in the image.

[0077] Specifically, when the knees of the human body are facing right, the orientation determination unit 15 determines that the human body is facing right. When the knees of the human body are facing left, the orientation determination unit 15 determines that the human body is facing left.

[0078] The knee orientation can be defined as the direction from an arbitrary point (e.g., midpoint) on a line segment connecting the key points of the waist and the foot toward the key point of the elbow. Based on this definition, the orientation determination unit 15 can determine the orientation of the knees of the human body on the image. In the example shown in FIG. 4, the orientation of the right knee is the direction d1 from an arbitrary point on the line segment connecting the right waist C12 and the right foot C16 toward the right knee C14. As can be seen from the figure, in this example, the orientation of the right knee is toward the left. Therefore, the orientation determination unit 15 determines that the human body is facing leftward.

[0079] - Determining the degree to which the human body is turned sideways - The orientation determining unit 15 determines the degree to which the human body is turned sideways on the image based on the magnitude of the actual measurement value relative to the reference value.

[0080] As described above, the "reference value" indicates the horizontal distance on the image between the first key point and the second key point when a human body having the height or the length of a predetermined part indicated in the reference information is captured in the image in a predetermined orientation. In this embodiment, the "predetermined orientation" refers to the orientation facing the camera.

[0081] As described above, the "actual measurement value" indicates the actual distance in the lateral direction between the first key point and the second key point of the human body on the image.

[0082] The first key point and the second key point are as described above.

[0083] The actual horizontal distance W between the first keypoint and the second keypoint on the image varies depending on the orientation of the human body. As shown in Fig. 6, when the human body H faces the camera directly (P = 0°) and when it faces 180° away (when the body is facing away from the camera, P = 180°), the actual horizontal distance W between the first keypoint and the second keypoint on the image is maximum. In this embodiment, the "predetermined orientation" is defined as the orientation facing the camera directly, and the distance W at this time is defined as the reference value.

[0084] As the human body H turns sideways from there, the distance W becomes smaller. When the human body turns sideways, the distance W becomes 0.

[0085] Based on this relationship, the orientation determination unit 15 determines the degree to which the human body is oriented sideways. For example, the orientation determination unit 15 can determine the degree to which the human body is oriented sideways based on a value R (= (actual measurement value) / (reference value)) that indicates the magnitude of the actual measurement value relative to a reference value. In this example, the larger the value of R, the smaller the degree to which the human body is oriented sideways, and the smaller the value of R, the greater the degree to which the human body is oriented sideways.

[0086] —Determining Which of Multiple Orientations the Person is Facing—First, based on the determination result of whether the person is facing forward or backward and the determination result of whether the person is facing right or left, the orientation determination unit 15 identifies which of the four categories in Fig. 5 the orientation of the person's body falls into. The four categories are "facing forward and left," "facing backward and left," "facing forward and right," and "facing backward and right."

[0087] Then, the orientation determination unit 15 determines which of the identified categories the orientation falls into based on the determination result of the degree of landscape orientation (the value of R described above).

[0088] A more detailed explanation will be given using the specific example of Fig. 6. In the example of Fig. 6, eight orientations are defined in 45° increments as the orientation of the human body on the image. A representative value of the angle P is indicated for each orientation. That is, eight orientations are defined with P = 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° as representative values. A numerical range that R (= (measured value) / (reference value)) can take when the human body is captured on the image in each orientation is associated with each orientation. For two orientations with P = 0° and 180° as representative values, R ≥ R １ The two orientations, P=90° and P=270°, are associated with each other. ２ The four orientations P=45°, 135°, 225°, and 315° are respectively associated with R ２ ≦R<R １ The information indicating the relationship shown in FIG.

[0089] For example, the orientation determination unit 15 determines that the orientation of the human body corresponds to "facing forward / left" of the four categories shown in FIG. 6 based on the determination results of whether the human body is facing forward or backward and whether the human body is facing right or left. As shown in FIG. 6, "facing forward / left" includes three types of orientations, with P = 0°, 45°, and 90° as representative values. The orientation determination unit 15 determines which of these three types of orientations the human body orientation corresponds to based on the value of R. Specifically, the orientation determination unit 15 determines which of the numerical ranges of R the value of R falls within, corresponding to each of the three types of orientations. The orientation determination unit 15 then determines the orientation that includes the value of R as the orientation of the human body in the image.

[0090] Next, an example of the processing flow of the processing device 10 will be described with reference to the flowchart of FIG.

[0091] First, the processing device 10 detects a plurality of key points of a human body included in an image (S10).

[0092] Next, the processing device 10 generates reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points detected in S10 (S11).

[0093] Next, the processing device 10 calculates a reference value indicating the height indicated by the reference information generated in S11, or the lateral distance on the image between the first key point and the second key point of the human body when a human body of a specified length is shown in the image in a specified orientation (S12).

[0094] Next, the processing device 10 calculates an actual measurement value indicating the lateral distance between the first key point and the second key point of the human body on the image based on the multiple key points detected in S10 (S13).

[0095] The processing device 10 then determines which of a plurality of orientations the human body is facing on the image based on the reference value calculated in S12, the actual measurement value calculated in S13, and the positional relationship on the image between the key points on the right side of the human body and the key points on the left side of the human body (S14). The orientation of the human body on the image is represented by an angle P ranging from 0° to 360°. Although not shown, the processing device 10 can output the determination result via a predetermined output device. Examples of the output device include, but are not limited to, a display, a projection device, a speaker, etc.

[0096] "Effects" According to the processing device 10 of the present embodiment, in a technology for determining the orientation of a human body based on data indicating the positions of multiple key points on a person, the orientation of the human body can be determined more precisely. Specifically, according to the processing device 10, the orientation of a human body on an image can be determined as multiple orientations represented by angles from 0° to 360°. According to the processing device 10 that can determine the orientation of a human body more precisely in this way, the versatility of the technology for determining the orientation of a human body based on data indicating the positions of multiple key points on a person is expanded.

[0097] Furthermore, the processing device 10 can determine the orientation of the human body on the image based on the results of determining whether the human body is facing forward or backward, whether the human body is facing right or left, and the degree to which the human body is facing sideways. With this processing device 10, the orientation of the human body on the image can be determined more precisely and with higher accuracy.

[0098] Furthermore, the processing device 10 can determine whether a human body is facing forward or backward on an image based on the relative positional relationship in the left-right direction on the image between the paired key points on the right side and the paired key points on the left side of the human body. With this processing device 10, it is possible to determine with high accuracy whether a human body is facing forward or backward on an image using relatively simple computer processing.

[0099] Furthermore, the processing device 10 can determine whether the human body is facing right or left on the image based on at least one of the "detection results of the key points on the right side and the key points on the left side," "the direction of the elbow," and "the direction of the knee." With this processing device 10, it is possible to determine with high accuracy whether the human body is facing right or left on the image using relatively simple computer processing.

[0100] Furthermore, the processing device 10 can determine the degree to which a human body is facing sideways in an image based on the magnitude of the actual measurement value relative to a reference value. The "reference value" is a theoretical value of the horizontal distance on an image between the first and second key points when a human body having a predetermined height or a predetermined length of a predetermined body part is shown in an image in a predetermined orientation. The "actual measurement value" is an actual measurement value of the horizontal distance on an image between the first and second key points of a human body having a predetermined height or a predetermined length of a predetermined body part. Such a processing device 10 can determine the degree to which a human body is facing sideways in an image with high accuracy using relatively simple computer processing.

[0101] Third Embodiment The processing device 10 of this embodiment detects a predetermined detection target included in an image in addition to detecting the orientation of a human body. Then, the processing device 10 determines the relationship between the human body and the predetermined detection target based on the detection result. This will be described in detail below.

[0102] 8 shows an example of a functional block diagram of the processing device 10 of this embodiment. As shown in the figure, the processing device 10 of this embodiment has a skeletal structure detection unit 11, a reference information generation unit 12, a reference value calculation unit 13, an actual measurement value calculation unit 14, an orientation determination unit 15, a detection unit 16, and a judgment unit 17.

[0103] The detection unit 16 detects a detection target included in the image, and outputs information indicating at least one of the position and orientation of the detected detection target on the image.

[0104] An "image" is an image in which multiple key points of a human body have been detected by the skeletal structure detection unit 11. That is, for one image, the process of detecting multiple key points of a human body by the skeletal structure detection unit 11 and the process of detecting a detection target by the detection unit 16 are performed.

[0105] The "detection target" can be determined depending on the usage situation of the processing device 10. For example, the detection target can be an object that is subject to human handling, such as a vending machine, digital signage, automatic doors, or display shelves.

[0106] The "position of the detected object on the image" is indicated, for example, by coordinates in a two-dimensional coordinate system set on the image. Alternatively, the detection unit 16 may generate a three-dimensional image by estimating depth information from the two-dimensional image. Then, the detection unit 16 may indicate the position of the detected object on the image by coordinates in a three-dimensional coordinate system set on the three-dimensional image. The generation of a three-dimensional image from a two-dimensional image can be realized using any technology.

[0107] The "orientation of the detected object on the image" refers to the orientation of the main surface of the detected object. There are various ways to define which surface of the detected object is the main surface, but for example, the surface facing a person can be defined as the main surface. In the case of a vending machine, for example, the surface on which products are displayed and where a coin slot and the like are located is defined as the main surface. In the case of digital signage, for example, the surface on which a display is located and information is displayed is defined as the main surface. In the case of an automatic door, for example, the surface on which the door opens and closes, through which users enter and exit, is defined as the main surface. In the case of a display shelf, for example, the surface facing the customer when products are displayed is defined as the main surface.

[0108] The orientation of the detection object on the image can be detected, for example, by a method similar to that for detecting the orientation of a human body on the image described in the second embodiment. That is, the orientation of the detection object on the image is represented by an angle S ranging from 0° to 360°. In this example, information indicating the relationship shown in FIG. 9 is stored in advance in the processing device 10. The detection unit 16 identifies the orientation of the detection object based on this information.

[0109] The detection unit 16 can determine whether the object to be detected faces forward or backward, or faces right or left, through image analysis. For example, the detection unit 16 can determine whether the object to be detected shown in an image faces forward or backward, or faces right or left, based on a learning model generated by machine learning using images of the object to be detected in various orientations as learning data.

[0110] Furthermore, the detection unit 16 can determine the degree to which the main surface of the detection object is oriented horizontally based on the aspect ratio T (= (horizontal length) / (vertical length)) of the main surface of the detection object on the image. The smaller the aspect ratio T, the smaller the degree to which the main surface of the detection object is oriented horizontally. And the larger the aspect ratio T, the greater the degree to which the main surface of the detection object is oriented horizontally.

[0111] In the example of Fig. 9, eight orientations are defined in 45° increments as the orientation of the detection object on the image. A representative value of the angle S is indicated for each orientation. That is, eight orientations are defined with S = 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° as representative values. A range of values ​​that the aspect ratio T can take when the detection object is captured in the image in each orientation is associated with each orientation. For the two orientations with S = 0° and 180° as representative values, T ≤ T １ The two orientations, T=90° and 270°, are associated with each other. ２ The four orientations, each of which has S=45°, 135°, 225°, and 315° as its representative value, are associated with T １ <T≦T ２ are associated.

[0112] For example, the detection unit 16 determines that the orientation of the detection object corresponds to "facing forward / left" of the four categories shown in FIG. 9 based on the determination results of whether the object faces forward or backward and whether the object faces right or left. As shown in FIG. 9, "facing forward / left" includes three orientations, with S=0°, 45°, and 90° as representative values. The detection unit 16 determines which of these three orientations the orientation of the detection object corresponds to based on the value of T. Specifically, the detection unit 16 determines which of the numerical ranges of T corresponding to each of the three orientations the value of T falls within. The detection unit 16 then determines the orientation that includes the value of T as the orientation of the detection object in the image.

[0113] In Fig. 9, eight types of orientations are defined in 45° increments, but this is not limiting. The shape of the detection object may be standardized. In such cases, the relationship shown in Fig. 9 can be specified in advance for each detection object and stored in the processing device 10.

[0114] 8 , the determination unit 17 determines the action of the human body relative to the detection target based on the orientation of the human body in the image and the position of the detection target in the image. For example, if the detection target is in the direction in which the human body is facing, the determination unit 17 can determine that "the human body is facing the detection target." This determination may be realized based on, for example, a three-dimensional image generated by estimating depth information from a two-dimensional image.

[0115] Furthermore, the determination unit 17 can determine the action of the human body toward the detection target based on the orientation of the human body and the orientation of the detection target in the image. For example, if the determination unit 17 determines that "the human body is facing the detection target," it can determine the angle from which the human body is looking toward the detection target based on the orientation of the human body and the orientation of the detection target in the image. If the difference between the orientation of the human body and the orientation of the detection target (the difference between angles P and S) is 180°, the determination unit 17 can determine that the human body is looking at the detection target directly in front. Then, based on this difference, the determination unit 17 can determine whether the human body is looking at the detection target directly in front or at an angle, and if so, the degree of obliqueness. The degree of obliqueness is calculated as the degree to which the difference deviates from 180°.

[0116] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the first and second embodiments.

[0117] The processing device 10 of this embodiment achieves the same effects as those of the first and second embodiments. Furthermore, the processing device 10 of this embodiment can determine the action of the human body relative to the detection target detected from the image based on the orientation of the human body in the image.

[0118] Here, an application example of the processing apparatus 10 of this embodiment will be described.

[0119] For example, the processing device 10 of this embodiment can be used in a vending machine. A camera is installed on or near the vending machine. The camera captures an image of a person standing in front of the vending machine. The processing device 10 then analyzes the image captured by the camera and determines the action of the person captured in the image relative to the vending machine. In this case, the camera and the processing device 10 are configured to be able to communicate with each other. The processing device 10 acquires the image generated by the camera in real time processing.

[0120] The processing device 10 then causes the vending machine to execute processing according to the determination result. For example, if it determines that the human body is facing the vending machine, the processing device 10 may cause the vending machine to output a voice message such as "Welcome." Also, if it determines that the human body is facing the vending machine, the processing device 10 may cause the vending machine to display predetermined information on a display provided on the vending machine. In this case, the processing device 10 and the vending machine are configured to be able to communicate with each other. The processing device 10 then sends an instruction to the vending machine to execute a predetermined process. The vending machine executes the predetermined process in accordance with the instruction.

[0121] It should be noted that similar processing can be performed using digital signage instead of vending machines.

[0122] As another application example, the processing device 10 of this embodiment can be used in an automatic door. A camera is installed around the automatic door. The camera captures an image of a person standing in front of the automatic door. The processing device 10 then analyzes the image captured by the camera and determines the action of the person captured in the image in relation to the automatic door. In this case, the camera and the processing device 10 are configured to be able to communicate with each other. The processing device 10 acquires the image generated by the camera in real time processing.

[0123] The processing device 10 then causes the automatic door to execute processing according to the determination result. For example, if it determines that a human body is located in front of the automatic door and that the human body is facing the automatic door, the processing device 10 may cause the automatic door to open the door. On the other hand, if it determines that a human body is located in front of the automatic door but is not facing the automatic door, the processing device 10 may not cause the automatic door to open the door. This configuration can prevent the inconvenience of opening the automatic door in response to the position of a person simply passing in front of the automatic door.

[0124] In this case, the processing device 10 and the automatic door are configured to be able to communicate with each other. The processing device 10 then sends instructions to the automatic door to execute a predetermined process (opening or closing the door). The automatic door executes the predetermined process in response to the instructions.

[0125] As another application example, the processing device 10 of this embodiment can be used in a store equipped with display shelves. Cameras are installed around the display shelves. The cameras capture images of people positioned in front of the display shelves. The processing device 10 then analyzes the images captured by the cameras and determines the actions of the people captured in the images with respect to the display shelves. In this case, the cameras and the processing device 10 are configured to be able to communicate with each other. The processing device 10 may acquire images generated by the cameras in real-time processing or in batch processing.

[0126] The processing device 10 can then generate market information based on the determination results. For example, the processing device 10 can count the number of people looking toward each display shelf. The processing device 10 can count the number of people looking toward each display shelf for each environment, such as by time of day, day of the week, month, or weather. The processing device 10 can also count the number of people of each attribute looking toward each display shelf based on the attributes of the people. The attributes include, but are not limited to, gender, age, nationality, etc. The processing device 10 can identify the attributes of the people through image analysis.

[0127] Fourth Embodiment A processing device 10 according to this embodiment identifies the action of a human body based on a time-series change in the orientation of the human body on an image. This will be described in detail below.

[0128] 10 shows an example of a functional block diagram of the processing device 10 of this embodiment. As shown in the figure, the processing device 10 of this embodiment has a skeletal structure detection unit 11, a reference information generation unit 12, a reference value calculation unit 13, an actual measurement value calculation unit 14, an orientation determination unit 15, and an identification unit 18.

[0129] The identification unit 18 identifies an action of the human body based on a time-series change in the orientation of the human body on the image. Features of the time-series change in the orientation of the human body during each action are linked to each of a plurality of actions and registered in advance in the processing device 10. The identification unit 18 identifies the action of the human body by detecting the features from the time-series change in the orientation of the human body on the image.

[0130] An example of an action is a behavior of looking around. A characteristic of a time series change in the orientation of a human body that is specific to this action is frequent changes in the orientation of the human body. This characteristic is defined, for example, as "the orientation changes occur a predetermined number of times or more within a predetermined time period," but is not limited to this.

[0131] Another example of an action is the behavior of looking in one direction. A characteristic of the time series change in the orientation of the human body that is specific to this action is the absence of a change in the orientation of the human body. This characteristic is defined, for example, as "no change in orientation occurs continuously for a predetermined period of time or more," but is not limited to this.

[0132] Another example of an action is a behavior of looking around while moving only the face without moving the torso. A characteristic of the time series change in the orientation of the human body specific to this action is frequent changes in the orientation of the face and no changes in the orientation of the torso. For example, this characteristic is defined as "the number of changes in the orientation of the face occurring within a predetermined time period is equal to or greater than a first predetermined number, and the number of changes in the orientation of the torso occurring within a predetermined time period is equal to or less than a second predetermined number," but is not limited to this.

[0133] In this example, the processing device 10 determines both the "face orientation" and the "torso orientation" as the orientation of the human body on the image. The face orientation can be determined by using the left ear C3 and the right ear C4 as the first and second key points described above. Furthermore, the torso orientation can be determined by using the left shoulder C5 and the right shoulder C6 (or the left hip C11 and the right hip C12) as the first and second key points.

[0134] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the first to third embodiments.

[0135] The processing device 10 of this embodiment achieves the same effects as those of the first to third embodiments. Furthermore, the processing device 10 of this embodiment can identify the actions of a human body based on time-series changes in the orientation of the human body in an image. For example, the processing device 10 can identify behaviors such as looking around, staring in one direction, and looking around while moving only the face without moving the torso.

[0136] Here, an application example of the processing device 10 of this embodiment will be described. For example, the processing device 10 of this embodiment can be used in a predetermined facility such as a store, a station, or an airport. A camera is installed in the facility. The camera captures images of people in the facility. The processing device 10 then analyzes the images captured by the camera and determines the actions of the people captured in the images. In this case, the camera and the processing device 10 are configured to be able to communicate with each other. The processing device 10 may acquire images generated by the camera in real time processing or in batch processing.

[0137] When the processing device 10 detects a human body performing a predetermined action, it notifies a predetermined person to be notified of the detection. The "predetermined action" may be, but is not limited to, a behavior of looking around, or a behavior of looking around while moving only the face without moving the torso, etc. The "person to be notified" may be a facility staff member, supervisor, security guard, etc. The processing device 10 notifies the person to be notified using any means such as email or an application.

[0138] <Modifications> Here, modifications applicable to all the embodiments will be described.

[0139] The reference value calculation unit 13 can calculate the reference value taking into consideration the attributes of the human body.

[0140] The "human body attributes" are those that can be identified by image analysis, such as gender, age, nationality, etc. The reference value calculation unit 13 identifies these attributes by image analysis.

[0141] As described in the second embodiment, the reference value calculation unit 13 calculates a reference value indicating the lateral distance on the image between the first key point and the second key point of the human body when the human body is shown in the image in a predetermined orientation, based on the height of the human body or the length of a predetermined part of the human body indicated in the reference information. Then, the reference value calculation unit 13 calculates the reference value based on a calculation model that calculates the reference value from the reference information.

[0142] In a modified example, the calculation model is generated for each of a plurality of categories of human bodies classified by the attributes. Each calculation model is optimized for the human body of the respective category. Then, the reference value calculation unit 13 calculates a reference value from the reference information based on the calculation model corresponding to the identified attribute of the human body.

[0143] In this modification, the same effects as those of the above embodiment are achieved. Furthermore, according to this modification, by calculating a reference value suited to each human body, it becomes possible to more accurately determine the orientation of the human body on the image.

[0144] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations may be adopted. The configurations of the above-described embodiments may be combined with each other, or some of the configurations may be replaced with other configurations. Furthermore, various modifications may be made to the configurations of the above-described embodiments without departing from the spirit of the invention. Furthermore, the configurations and processes disclosed in the above-described embodiments and modified examples may be combined with each other.

[0145] In addition, in the flowcharts used in the above explanation, multiple steps (processes) are described in order. However, the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. Furthermore, each of the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0146] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: 1. A processing device comprising: a skeletal structure detection means for detecting a plurality of key points of a human body included in an image; a reference information generation means for generating reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points; a reference value calculation means for calculating, based on the reference information, a reference value indicating the horizontal distance on the image between a first key point and a second key point of the human body when the human body is shown in the image in a predetermined orientation; an actual measurement calculation means for calculating, based on the plurality of key points, an actual measurement value indicating the horizontal distance on the image between the first key point and the second key point of the human body; and an orientation determination means for determining which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is, based on the reference value, the actual measurement value, and a positional relationship on the image between the key point on the right side of the human body and the key point on the left side of the human body. The processing device according to 1, wherein the orientation determination means determines whether the human body is facing forward or backward, whether the human body is facing right or left, and the degree to which the human body is facing sideways, and determines the orientation of the human body on the image based on the determination results. 3. The processing device according to 2, wherein the orientation determination means determines whether the human body is facing forward or backward on the image based on the relative positional relationship in the left-right direction on the image between the key points on the right side and the key points on the left side that are paired with each other. 4. The processing device according to 2 or 3, wherein the orientation determination means determines whether the human body is facing right or left on the image based on at least one of: whether the key points on the right side and the key points on the left side of the human body are detected in detecting the multiple key points; the orientation of elbows indicated by the multiple key points; and the orientation of knees indicated by the multiple key points. 5. The processing device according to any of 2 to 4, wherein the orientation determination means determines the degree to which the human body is facing sideways on the image based on the magnitude of the actual measurement value relative to the reference value.6. The processing device according to any one of 1 to 5, further comprising: a detection means for detecting a detection object included in the image; and a determination means for determining an action of the human body relative to the detection object based on the orientation of the human body on the image and the position of the detection object on the image. 7. The processing device according to 6, wherein the detection means identifies the orientation of the detection object on the image, and the determination means determines the action of the human body relative to the detection object based on the orientation of the human body and the orientation of the detection object. 8. The processing device according to any one of 1 to 7, further comprising an identification means for identifying the action of the human body based on a time series change in the orientation of the human body on the image. 9. A processing method in which one or more computers detect a plurality of key points on a human body included in an image; generate reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points; calculate a reference value indicating the horizontal distance on the image between a first key point and a second key point on the human body when the human body is shown in the image in a predetermined orientation based on the reference information; calculate an actual measurement value indicating the horizontal distance on the image between the first key point and the second key point on the human body based on the plurality of key points; and determine which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is based on the reference value, the actual measurement value, and a positional relationship on the image between the key point on the right side of the human body and the key point on the left side of the human body.10. A program causing a computer to function as: a skeletal structure detection means for detecting a plurality of key points of a human body included in an image; a reference information generation means for generating reference information indicating the height of the human body or the length of a predetermined part of the human body based on the plurality of key points; a reference value calculation means for calculating, based on the reference information, a reference value indicating the lateral distance on the image between a first key point and a second key point of the human body when the human body is shown in the image in a predetermined orientation; an actual measurement calculation means for calculating, based on the plurality of key points, an actual measurement value indicating the lateral distance on the image between the first key point and the second key point of the human body; and an orientation determination means for determining which of a plurality of orientations represented by angles between 0° and 360° the orientation of the human body on the image is based on the reference value, the actual measurement value, and the positional relationship on the image between the key point on the right side of the human body and the key point on the left side of the human body.

[0147] This application claims priority based on Japanese Patent Application No. 2023-064775, filed April 12, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0148] REFERENCE SIGNS LIST 10 Processing device 11 Skeletal structure detection unit 12 Reference information generation unit 13 Reference value calculation unit 14 Actual measurement value calculation unit 15 Orientation determination unit 16 Detection unit 17 Determination unit 18 Identification unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. a skeletal structure detection means for detecting a plurality of key points of a human body included in an image; a reference information generating means for generating reference information indicating the height of the human body or the length of a predetermined part of the human body based on a plurality of the key points; a reference value calculation means for calculating a reference value indicating a lateral distance on the image between a first key point and a second key point of the human body when the human body is photographed in the image in a predetermined orientation based on the reference information; an actual measurement value calculation means for calculating an actual measurement value indicating a lateral distance between the first key point and the second key point of the human body on the image based on the plurality of key points; an orientation determination means for determining which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is based on the reference value, the actual measurement value, and a positional relationship on the image between the key points on the right side of the human body and the key points on the left side of the human body; A processing device having:

2. 2. The processing device according to claim 1, wherein the orientation determination means determines whether the human body is facing forward or backward, whether the human body is facing right or left, and the degree to which the human body is facing sideways, and determines the orientation of the human body on the image based on the results of the determination.

3. 3. The processing device according to claim 2, wherein the orientation determination means determines whether the human body is facing forward or backward on the image based on the relative positional relationship in the left-right direction of the key points on the right side and the key points on the left side of the human body that are paired with each other on the image.

4. The orientation determination means Whether the key points on the right side of the human body and the key points on the left side of the human body are detected in the detection of the plurality of key points; The elbow orientation indicated by the plurality of key points; and the orientation of the knee indicated by a plurality of said key points; The processing device according to claim 2 , wherein the processing device determines whether the human body is facing right or left on the image based on at least one of the following:

5. 3. The processing device according to claim 2, wherein the orientation determining means determines the degree to which the human body is facing sideways on the image based on the magnitude of the actual measurement value relative to the reference value.

6. a detection means for detecting a detection target included in the image; a determining means for determining an action of the human body relative to the detection object based on the orientation of the human body on the image and the position of the detection object on the image; The processing apparatus of claim 1 further comprising:

7. the detection means identifies the orientation of the detection object on the image; The processing device according to claim 6 , wherein the determining means determines an action of the human body relative to the detection object based on the orientation of the human body and the orientation of the detection object.

8. The processing device according to claim 1 , further comprising: an identifying unit that identifies an action of the human body based on a time-series change in the orientation of the human body on the image.

9. One or more computers Detect multiple key points of the human body in the image, generating reference information indicating a height of the human body or a length of a predetermined part of the human body based on a plurality of the key points; calculating a reference value indicating a lateral distance on the image between a first key point and a second key point of the human body when the human body is photographed in a predetermined orientation on the image based on the reference information; calculating an actual measurement value indicating a lateral distance between the first key point and the second key point of the human body on the image based on the plurality of key points; A processing method for determining which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is based on the reference value, the actual measurement value, and the positional relationship on the image between the key points on the right side of the human body and the key points on the left side of the human body.

10. Computer, a skeletal structure detection means for detecting a plurality of key points of a human body included in an image; a reference information generating means for generating reference information indicating the height of the human body or the length of a predetermined part of the human body based on a plurality of the key points; a reference value calculation means for calculating, based on the reference information, a reference value indicating a lateral distance on the image between a first key point and a second key point of the human body when the human body is shown in the image in a predetermined orientation; an actual measurement value calculation means for calculating an actual measurement value indicating a lateral distance between the first key point and the second key point of the human body on the image based on the plurality of key points; an orientation determination means for determining which of a plurality of orientations represented by angles from 0° to 360° the orientation of the human body on the image is based on the reference value, the actual measurement value, and a positional relationship on the image between the key points on the right side of the human body and the key points on the left side of the human body; A program that functions as a