Video processing system

The video processing system normalizes target poses by applying joint rotations to a base model, addressing the mismatch issue caused by size and shape variations, resulting in more accurate behavioral feature extraction.

JP2024166910AActive Publication Date: 2024-11-29TOYOTA JIDOSHA KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023083335
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-29
Estimated Expiration
2043-05-19

AI Technical Summary

Technical Problem

Existing techniques for estimating the pose of targets in videos face challenges due to variations in size, shape, and orientation, leading to mismatched behavioral features even for the same behavior, which affects the accuracy of applications using these features.

Method used

A video processing system that normalizes the pose of targets by applying joint rotations to a base target model, converting the base pose to a normalized pose, and extracting behavioral features from this normalized pose, thereby excluding the influence of size, shape, and orientation.

Benefits of technology

The normalization process enhances the robustness of behavioral features, ensuring they accurately reflect the target's behavior, thereby improving the accuracy of applications that utilize these features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166910000001_ABST
    Figure 2024166910000001_ABST
Patent Text Reader

Abstract

To acquire action feature amounts appropriately reflecting actions of a target shown in a video.SOLUTION: A video processing system is applied to a video showing a target. The video processing system estimates a pose of the target shown in the video as an original pose. The video processing system acquires information on joint rotation of a predetermined joint of the target having the original pose. Furthermore, the video processing system converts a base pose of a base target model into a normalized pose of the target by applying the joint rotation to the predetermined joint of the base target model having the base pose. Then, the video processing system extracts a series of feature amounts of the normalized pose of the target in time series as action feature amounts of the target.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to image processing techniques that are applied to images showing targets. [Background technology]

[0002] Patent Document 1 discloses a three-dimensional human posture estimation device that estimates the three-dimensional posture of a photographed person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-092528 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a known technology for estimating the posture of a target (e.g., a person) captured in a video. Features extracted from a series of postures of a target over time are behavioral features that reflect the target's actions. The behavioral features of a target are useful, for example, for recognizing the target's actions and detecting abnormal actions.

[0005] However, the size, shape, orientation, etc. of targets captured in the video are various. When the size, shape, or orientation of the targets are different, the extracted behavioral features may not match sufficiently even for the same behavior. In order to improve the accuracy of applications that use behavioral features, it is desirable that the behavioral features extracted for the same behavior match as much as possible.

[0006] One object of the present disclosure is to provide a technology capable of acquiring behavioral features that appropriately reflect the behavior of a target captured in a video. [Means for solving the problem]

[0007] One aspect of the present disclosure relates to an image processing system applied to an image showing a target. The video processing system includes one or more processors. The one or more processors estimate a posture of a target captured in an image as an original posture, and obtain information on joint rotation of a predetermined joint of the target having the original posture. Additionally, the one or more processors transform the base pose of the base target model to a normalized pose of the target by applying joint rotations to predetermined joints of the base target model having the base pose. Then, the one or more processors extract a series of feature amounts of the normalized postures of the target in time series as behavior feature amounts of the target. Effect of the Invention

[0008] According to the present disclosure, the normalized posture of the target is obtained by normalizing the original posture of the target shown in the video. The normalized posture of the target excludes the influence of the size, shape, and the like of the target. Based on such a normalized posture, the behavioral feature of the target is extracted. Therefore, behavioral feature for the same behavior is more likely to match. In other words, behavioral feature that appropriately reflects the behavior of the target is obtained. As a result, the accuracy of the application that uses the behavioral feature is improved. [Brief description of the drawings]

[0009] [Figure 1] 1 is a conceptual diagram for explaining an overview of a video processing system according to an embodiment. [Diagram 2] 1 is a block diagram illustrating an example of a functional configuration of a video processing system according to an embodiment. [Diagram 3] 11A and 11B are conceptual diagrams for explaining an example of normalization processing from a spatial viewpoint according to an embodiment. [Figure 4] FIG. 11 is a conceptual diagram for explaining an example of a normalization process from a time perspective according to the embodiment. [Diagram 5] FIG. 11 is a conceptual diagram illustrating an example of a standardization process according to the embodiment. [Figure 6] 1 is a block diagram illustrating an example of a functional configuration of a video processing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1. Overview of the video processing system 1 is a conceptual diagram for explaining an overview of a video processing system 100 according to the present embodiment. The video processing system 100 acquires video VID captured by a camera CAM, and performs various processes on the video VID.

[0011] In particular, the video processing system 100 is applied to a video VID in which a target TG appears. In this embodiment, the target TG is a moving object having a joint. Examples of such a target TG include a person, a part of a person (e.g., a hand), a robot, and the like. In the following description, the case where the target TG is a person will be mainly considered. However, the target TG is not limited to a person.

[0012] As an example, consider recognizing the action of a target TG shown in a video VID. The video processing system 100 estimates the pose of the target TG shown in each of a series of images constituting the video VID. A series of poses arranged in time series is hereinafter referred to as a "pose sequence." The video processing system 100 extracts features of the posture sequence of the target TG. The features extracted from the posture sequence of the target TG reflect the action of the target TG. Such features are hereinafter referred to as "action feature FEA." A database showing the correspondence between types of actions and action feature FEA is created in advance. The video processing system 100 can recognize (classify, identify) the action of the target TG by checking the action feature FEA of the target TG against the database.

[0013] The use of the behavior feature FEA is not limited to behavior recognition. For example, the behavior of the target TG shown in the video VID can be analyzed. For example, abnormal behavior of the target TG can be detected by comparing the distribution of the behavior feature FEA representing normal behavior with the behavior feature FEA of the target TG. As another example, the degree of proficiency of the behavior of the target TG can be estimated by comparing the distribution of the behavior feature FEA representing a well-trained behavior with the behavior feature FEA of the target TG.

[0014] As yet another example, by performing matching using the behavior feature amount FEA of the target TG, it is possible to search for other targets performing the same behavior as the target TG from within the video database. As yet another example, by performing matching using the behavior feature amount FEA of the target TG, it is possible to search for other videos in which other targets performing the same behavior as the target TG are shown from within the video database.

[0015] However, the size, shape, view, etc. of the target TG shown in the video VID vary. If the size, shape, or view of the target TG is different, the extracted behavioral feature FEA may not match sufficiently even if the behavior is the same.

[0016] For example, in FIG. 1, the video VID-1 captured by the camera CAM-1 shows a running target TG-1. The video VID-2 captured by the camera CAM-2 shows a running target TG-2. Both targets TG-1 and TG-2 are running. That is, the "behavior" of targets TG-1 and TG-2 is the same. However, the size, shape, orientation, etc. of targets TG-1 and TG-2 captured in the videos VID-1 and VID-2 are different. Therefore, there is a possibility that the behavior feature FEA-1 of target TG-1 captured in the video VID-1 and the behavior feature FEA-2 of target TG-2 captured in the video VID-2 do not sufficiently match. That is, even though the "behavior" of targets TG-1 and TG-2 is the same, there is a possibility that their respective behavior feature FEA-1 and FEA-2 do not sufficiently match.

[0017] If the action feature FEA for the same action does not match sufficiently, the accuracy of the application using the action feature FEA may decrease. In order to improve the accuracy of the application using the action feature FEA, it is desirable that the action feature FEA extracted for the same action matches as much as possible. In other words, it is desirable to improve the robustness of the action feature FEA with respect to the size, shape, orientation, etc. of the target TG.

[0018] Therefore, according to the present embodiment, in order to improve the robustness of the behavior feature amount FEA, "normalization" of the posture of the target TG is performed as a pre-processing for extracting the behavior feature amount FEA. Hereinafter, processing related to the normalization processing according to the present embodiment will be described in detail.

[0019] 2. Processing related to standardization 2 is a block diagram showing an example of a functional configuration of a video processing system 100 according to this embodiment. The video processing system 100 includes a posture estimation unit 110, a standardization processing unit 120, and a feature extraction unit .

[0020] The video processing system 100 acquires a video VID captured by a camera CAM. The video VID includes a series of images (frames). First, the video processing system 100 detects a target TG appearing in each image included in the video VID. The detection of the target TG can be achieved by a well-known object detection technique. A bounding box represents the position of the target TG detected in the image.

[0021] The posture estimation unit 110 performs posture estimation processing to estimate the three-dimensional posture of the target TG based on an image of a portion of the target TG. A partial image surrounded by each bounding box in an image corresponds to the image of the target TG. The posture estimation unit 110 extracts key points from the partial image by using a posture estimation model based on machine learning, and estimates the three-dimensional posture of the target TG. Note that the posture estimation processing is a well-known technique, and the method thereof is not particularly limited. For example, TransPose is used in the posture estimation processing.

[0022] For convenience, the three-dimensional posture of the target TG estimated by the posture estimation unit 110 is called the "original posture PSO". If the action feature amount FEA is directly extracted from this original posture PSO, the above-mentioned problem occurs. Therefore, the standardization processing unit 120 executes a standardization process for standardizing the original posture PSO of the target TG as a pre-processing of the feature amount extraction. The standardization process can also be called re-targeting. The details of the standardization process in the standardization processing unit 120 will be described later. The three-dimensional posture obtained by standardizing the original posture PSO is hereinafter called the "normalized posture PSN".

[0023] The feature extraction unit 130 extracts the action feature FEA of the target TG based on the normalized posture PSN instead of the original posture PSO. More specifically, the feature extraction unit 130 arranges the normalized posture PSN of the target TG in a time series. The normalized posture sequence is a series of normalized postures PSN arranged in a time series. The feature extraction unit 130 extracts the feature of the normalized posture sequence of the target TG as the action feature FEA of the target TG. For example, known ST-GCNs (Spatial-Temporal Graph Convolutional Networks) are used for such feature extraction processing.

[0024] The following is a detailed explanation of the normalization process performed by the normalization processing unit 120. As will be explained below, two types of normalization process are possible: normalization process from a spatial perspective and normalization process from a temporal perspective.

[0025] 2-1. Normalization from a spatial perspective As shown in FIG. 2, the standardization processing unit 120 includes a joint rotation acquisition unit 121, a base target model 122, and a model processing unit 123.

[0026] The joint rotation acquisition unit 121 acquires information on the original posture PSO of the target TG. The target TG has one or more joints JTi. At least a part of the key points extracted in the posture estimation process corresponds to the joints JTi. i is a joint identifier and takes a value of 1 or more. The information on the original posture PSO of the target TG includes three-dimensional arrangement information of the joints JTi of the target TG. Based on the three-dimensional arrangement information of the joints JTi, information on the joint rotation Ri of a given joint JTi can be acquired. That is, the joint rotation acquisition unit 121 can acquire information on the joint rotation Ri of a given joint JTi of the target TG having the original posture PSO. The joint rotation Ri is also called a joint angle. The joint rotation Ri of the joint JTi of the target TG does not depend on the size, shape, and orientation of the target TG.

[0027] The base target model 122 is a common model of the target TG. The base target model 122 has a base posture. In the example shown in Fig. 2, the base target model 122 is a T-Pose model having a T-pose as a base posture. The base target model 122 also has a predetermined joint JTi.

[0028] The model processing unit 123 acquires information on the joint rotation Ri of a predetermined joint JTi of the target TG from the joint rotation acquisition unit 121. Then, the model processing unit 123 applies the joint rotation Ri of the predetermined joint JTi of the target TG to the predetermined joint JTi of the base target model 122 having the base posture. By applying the joint rotation Ri to the predetermined joint JTi of the base target model 122, the base posture of the base target model 122 is converted. The original posture PSO of the target TG excluding the size and shape of the target TG is reflected in the posture of the base target model 122 after the conversion. Therefore, the posture of the base target model 122 after the conversion becomes the normalized posture PSN of the target TG from which the influence of the size and shape of the target TG is removed. That is, the model processing unit 123 converts the base posture of the base target model 122 to the normalized posture PSN of the target TG by applying the joint rotation Ri to the base target model 122.

[0029] Furthermore, the model processing unit 123 may align the orientation of the normalized attitude PSN to a "predetermined orientation". The predetermined orientation is specified in advance. Since the base target model 122 is a virtual model, its orientation can be freely changed. For example, the orientation of the base target model 122 is expressed by a quaternion. By making the quaternion of the base target model 122 a predetermined quaternion, the orientation of the normalized attitude PSN can be aligned to the predetermined orientation. That is, the model processing unit 123 converts the base attitude of the base target model 122 to the normalized attitude PSN of the target TG so that the orientation of the normalized attitude PSN becomes the predetermined orientation.

[0030] FIG. 3 shows an example of normalization processing from a spatial point of view. Target TG-1 is shown in image VID-1, and target TG-2 is shown in image VID-2. Targets TG-1 and TG-2 have different sizes, shapes, and orientations. Based on the original pose PSO-1 of target TG-1, information on joint rotation Ri-1 of target TG-1 is obtained. The joint rotation Ri-1 is applied to the base target model 122 to obtain a normalized pose PSN-1 of target TG-1. Furthermore, the normalized pose PSN-1 of target TG-1 is adjusted to be in a "predetermined orientation". The same is true for target TG-2.

[0031] The normalized attitude PSN-1 of the target TG-1 obtained in this manner eliminates the effects of the size, shape, and orientation of the target TG-1. Similarly, the normalized attitude PSN-2 of the target TG-2 eliminates the effects of the size, shape, and orientation of the target TG-2. In other words, robustness to the size, shape, and orientation of the target TG is improved.

[0032] As described above, the normalization processing unit 120 normalizes the original posture PSO of the target TG from a spatial viewpoint to obtain the normalized posture PSN of the target TG. Each joint JTi in the normalized posture PSN has information on the position (x, y, z) and time (t).

[0033] 2-2. Normalization from a time perspective As an example, consider a case where the target TG is a person and the action is running. Some people run fast and some people run slow. However, from the viewpoint of the action feature amount FEA, it is preferable to regard fast running and slow running as the same "running". In other words, it is preferable to make the action feature amount FEA for fast running and the action feature amount FEA for slow running as consistent as possible. For this purpose, normalization processing is performed from a time viewpoint.

[0034] FIG. 4 is a conceptual diagram for explaining an example of normalization processing from a time perspective. In the following description, the "shift amount SFT" is the amount of change in position of a predetermined joint JT of a target TG between different timings on a time series. For example, the shift amount SFT is the amount of change in position of a predetermined joint JT of a target TG between timings separated by a predetermined time. As another example, the shift amount SFT may be the amount of change in position of a predetermined joint JT of a target TG between consecutive timings. Since the joint JT in the normalized posture PSN has information on the position (x, y, z) and time (t), the shift amount SFT can be calculated based on this information.

[0035] The normalized posture sequence is a series of normalized postures PSN arranged in a time series. The normalization processing unit 120 calculates a shift amount SFT of a predetermined joint JT of the target TG based on the normalized posture sequence of the target TG. Furthermore, the normalization processing unit 120 inversely adjusts the normalized posture sequence so that the shift amount SFT becomes a predetermined reference amount SFT0. That is, the normalization processing unit 120 normalizes the shift amount SFT to the predetermined reference amount SFT0.

[0036] For example, in FIG. 4, a shift amount SFT-2 for the target TG-2 is calculated based on the position of the joint JT of the target TG-2 at time t2a and the position of the joint JT of the target TG-2 at time t2b. The calculated shift amount SFT-2 is larger than a predetermined reference amount SFT0 (SFT-2>SFT0). Therefore, a time t2x is searched for such that the shift amount SFT-2 from time t2a to time t2x matches the predetermined reference amount SFT0. Such a time t2x can be searched based on the positions of the joint JT at times t2a and t2b. Then, a new normalized posture PSN is inserted at time t2x. The position of the joint JT in the normalized posture PSN at time t2x can be calculated by linear interpolation based on the positions of the joint JT at times t2a and t2b. In this way, the normalized posture sequence of the target TG-2 is adjusted so that the shift amount SFT-2 for the target TG-2 matches the predetermined reference amount SFT0. This corresponds to "upsampling".

[0037] On the one hand, in FIG. 4, the shift amount SFT-1 for the target TG-1 is smaller than a predetermined reference amount SFT0 (SFT-1 < SFT0). In this case, "downsampling" is performed. The normalized posture sequence of the target TG-1 is adjusted so that the shift amount SFT-1 for the target TG-1 matches the predetermined reference amount SFT0.

[0038] The normalization processing unit 120 performs the above processing for each joint JT of the target TG, and normalizes the shift amount SFT to a predetermined reference amount SFT0.

[0039] As a modification, the normalization processing unit 120 may calculate the average value of the shift amounts SFT calculated for each joint JT of the target TG. It can be said that the average value is the shift amount SFT for the entire target TG. The shift amount SFT for the entire target TG corresponds to the speed of the entire target TG. The normalization processing unit 120 adjusts the normalized posture sequence of the target TG so that the shift amount SFT of the target TG becomes the predetermined reference amount SFT0.

[0040] FIG. 5 shows an example of the normalization process. The original posture PSO-1 of the target TG-1 is normalized to obtain the normalized posture PSN-1. Similarly, the original posture PSO-2 of the target TG-2 is normalized to obtain the normalized posture PSN-2. In the original postures PSO-1 and PSO-2, the sizes, shapes, orientations, and speeds of the targets TG-1 and TG-2 are different, but in the normalized postures PSN-1 and PSN-2, they are the same. That is, the influence of the size, shape, orientation, and speed of the target TG is eliminated from the normalized posture PSN of the target TG.

[0041] 2-4. Effects As described above, according to this embodiment, the normalized posture PSN of the target TG is obtained by normalizing the original posture PSO of the target TG shown in the video VID. The effects of the size, shape, orientation, and speed of the target TG are eliminated from the normalized posture PSN of the target TG. The behavior feature amount FEA of the target TG is extracted based on such a normalized posture PSN. Therefore, the robustness of the behavior feature amount FEA with respect to the size, shape, orientation, and speed of the target TG is improved. In other words, the behavior feature amount FEA for the same behavior is more likely to match. That is, the behavior feature amount FEA that appropriately reflects the behavior of the target TG is obtained. As a result, the accuracy of the application using the behavior feature amount FEA is improved.

[0042] Note that the effect can be obtained by only performing the normalization process from a spatial viewpoint described in Section 2-1 above. The normalization process from a temporal viewpoint is optional. Also, even if the orientation of the normalized posture PSN is not aligned to a predetermined orientation, the effect of the size and shape of the target TG is eliminated, so that a certain degree of effect can be obtained. Further effect can be obtained if the orientation of the normalized posture PSN is aligned to a predetermined orientation.

[0043] 3. Example of a video processing system configuration 6 is a block diagram showing an example of the configuration of a video processing system 100 according to this embodiment. The video processing system 100 includes one or more processors 101 (hereinafter simply referred to as "processor 101"), one or more storage devices 102 (hereinafter simply referred to as "storage device 102"), and an interface 103. The processor 101 executes various processes. For example, the processor 101 includes a CPU (Central Processing Unit). The storage device 102 stores various information required for the processes. Examples of the storage device 102 include a hard disk drive (HDD), a solid state drive (SSD), a volatile memory, a non-volatile memory, and the like. The interface 103 includes a network interface and a user interface.

[0044] The computer program 104 is a computer program for video processing according to the present embodiment. The computer program 104 is stored in the storage device 102. The computer program 104 may be recorded in a computer-readable recording medium. The computer program 104 is executed by the processor 101. The functions of the video processing system 100 may be realized by cooperation between the processor 101 executing the computer program 104 and the storage device 102.

[0045] In addition, various databases 105 are stored in the storage device 102 .

[0046] The processor 101 collects video VID captured by one or more cameras CAM via an interface 103. The database 105 includes a video database relating to the collected video VID.

[0047] The processor 101 also has the functions of the posture estimation unit 110, the normalization processing unit 120, and the feature extraction unit 130 shown in Fig. 2. The processor 101 extracts a behavior feature amount FEA of the target TG shown in the video VID. The database 105 includes a feature amount database related to the extracted behavior feature amount FEA. The feature amount database may indicate a correspondence relationship between the type of behavior and the behavior feature amount FEA.

[0048] The processor 101 can use the behavior feature amount FEA for various purposes. For example, the processor 101 can recognize (classify, identify) the behavior of the target TG by checking the behavior feature amount FEA of the target TG against a feature amount database.

[0049] As another example, the processor 101 may analyze the behavior of the target TG shown in the video VID. For example, by comparing the distribution of the behavior feature amount FEA representing normal behavior with the behavior feature amount FEA of the target TG, abnormal behavior of the target TG can be detected. As another example, by comparing the distribution of the behavior feature amount FEA representing a well-accustomed behavior with the behavior feature amount FEA of the target TG, the degree of proficiency of the behavior of the target TG can be estimated.

[0050] As yet another example, the processor 101 may search for other targets behaving in the same manner as the target TG from the video database by performing matching using the behavior feature amount FEA of the target TG. As yet another example, the processor 101 may search for other videos in which other targets behaving in the same manner as the target TG are captured from the video database by performing matching using the behavior feature amount FEA of the target TG. [Explanation of symbols]

[0051] 100...image processing system, 110...posture estimation unit, 120...standardization processing unit, 121...joint rotation acquisition unit, 122...base target model, 123...model processing unit, 130...feature extraction unit, FEA...action feature, PSN...standardized posture, PSO...original posture, VID...image

Claims

1. A video processing system applied to a video showing a target, comprising: One or more processors; The one or more processors: Estimating the pose of the target shown in the image as an original pose; Acquire joint rotation information of a predetermined joint of the target having the original pose; Transforming the base pose of the base target model to a normalized pose of the target by applying the joint rotations to the predetermined joints of a base target model having a base pose; A series of normalized posture features of the target in a time series are extracted as behavior features of the target. It was configured as Video processing system.

2. 2. The image processing system according to claim 1, The one or more processors are further configured to transform the base pose of the base target model to the normalized pose of the target such that the orientation of the normalized pose is a predetermined orientation. Video processing system.

3. 3. The image processing system according to claim 2, The one or more processors are further configured to adjust the series of normalized postures of the target on the time series such that a position change amount of the predetermined joint between different timings on the time series becomes a reference amount. Video processing system.

4. 3. The image processing system according to claim 2, The one or more processors are further configured to adjust the series of normalized attitudes of the target on the time series such that a position change amount of the target between different timings on the time series becomes a reference amount. Video processing system.

5. A video processing system according to any one of claims 1 to 4, The one or more processors further perform a first process based on the behavioral feature amount; The first process includes: Recognizing the behavior of the target captured in the video; analyzing the behavior of the target captured in the video; detecting abnormal behavior of the target in the video; Searching for other targets engaging in the same behavior as the target; and Searching for other footage showing other targets engaging in the same activities as said target Contains at least one of Video processing system.

Citation Information

Patent Citations

  • Skeleton point behavior identification system based on shift graph convolutional neural network and identification method thereof

    CN111582220A

  • Joint rotation inferences based on inverse kinematics

    WO2021186223A1

  • Image processing device, image processing method, and program

    WO2021255846A1

  • Three-dimensional person attitude estimation apparatus, method, and program

    JP2022092528A