Information processing device

The information processing device uses multiple camera integration and feature point analysis to track and generate movement history data, addressing the limitations of single-camera systems by enhancing data accuracy and privacy in commercial facility tracking.

JP7948882B2Active Publication Date: 2026-11-11NTT DOCOMO INC
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2023061394
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-04-05
Publication Date
2026-11-11
Estimated Expiration
2043-04-05

AI Technical Summary

Technical Problem

Existing systems struggle to accurately acquire detailed data on the behavior of specific objects within commercial facilities using images from a single camera, limiting the ability to track and analyze movements effectively.

Method used

An information processing device that integrates images from multiple cameras with different shooting ranges, using feature point detection, color information, and posture analysis to identify and track individuals across multiple images, generating movement history information based on time and camera data.

Benefits of technology

Enables accurate tracking and analysis of individual movements within facilities, providing detailed movement history data without relying on facial recognition, thus preserving privacy and enhancing data accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007948882000001
    Figure 0007948882000001
  • Figure 0007948882000002
    Figure 0007948882000002
  • Figure 0007948882000003
    Figure 0007948882000003
Patent Text Reader

Abstract

To provide an information processing device capable of suitably obtaining data related to the behavior of a specific object.SOLUTION: An information processing device 10 according to one embodiment comprises: an acquisition unit 11 which acquires association information for which images captured by each of multiple cameras C with photographic ranges different from each other, time information indicating the time at which the images are taken, and camera information to identify the cameras C which have taken the images are associated with each other; an analysis unit 12 which extracts multiple images including specific objects presumed to be the same as a common subject by analyzing the images included in the association information acquired by the acquisition unit 11; and a generation unit 13 which generates movement history information indicating the time change in the position of the specific object based on the time information and the camera information associated with each of the multiple images extracted by the analysis unit 12.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One aspect of the present invention relates to an information processing apparatus.

Background Art

[0002] Patent Document 1 discloses a method for identifying an object by: detecting a group of feature points corresponding to a plurality of sites from an image obtained by imaging an object (human body) having the plurality of sites (joints), extracting a feature amount corresponding to each feature point from the image, and comparing the extracted feature amount with a pre-registered feature amount of the object.

Prior Art Literature

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] For example, in commercial facilities and the like, there is a need to acquire data related to the behavior of a specific object (for example, a person such as a customer) (for example, data indicating the movement history of a user who has entered the commercial facility). However, it is difficult to acquire the aforementioned detailed data related to the behavior of a specific object only by identifying an object included in an image captured by a single camera installed at a specific location (for example, an entrance of a facility) using the technique as disclosed in Patent Document 1 described above.

[0005] Accordingly, one aspect of the present invention aims to provide an information processing apparatus capable of suitably acquiring data related to the behavior of a specific object.

Means for Solving the Problem

[0006] An information processing device according to one aspect of the present invention includes: an acquisition unit that acquires association information relating images taken by each of a plurality of cameras having different shooting ranges, time information indicating the time the images were taken, and camera information for identifying the camera that took the images; an analysis unit that identifies a plurality of images that include a specific object presumed to be the same as a common subject by analyzing the images included in the association information acquired by the acquisition unit; and a generation unit that generates movement history information indicating the time change of the position of a specific object based on the time information and camera information associated with each of the plurality of images identified by the analysis unit.

[0007] In an information processing device relating to one aspect of this disclosure, multiple images containing a common subject (specific object) are extracted by analyzing each image captured by each of multiple cameras. Then, based on the time information and camera information associated with each of these multiple images, movement history information (i.e., the time change of the specific object's position) is generated, indicating at what time and where (i.e., the location corresponding to the camera's shooting range identified by the camera information) the specific object was. Therefore, according to the above configuration, data concerning the behavior of the specific object (i.e., movement history information) can be suitably obtained. [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to provide an information processing device that can suitably acquire data relating to the behavior of a specific object. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the functional configuration of an information processing device according to one embodiment. [Figure 2] This figure shows an example of association information. [Figure 3] This figure shows an example of the functional configuration of the analysis unit. [Figure 4] This figure shows an example of skeletal information containing multiple feature points. [Figure 5]This diagram visually represents an example of color information (color histogram) corresponding to a single feature point. [Figure 6] This figure shows an example of posture determination processing. [Figure 7] This is a diagram showing an example of person information. [Figure 8] This diagram illustrates the effectiveness of using multiple postures to determine identity. [Figure 9] A flowchart illustrating an example of the operation of an information processing device. [Figure 10] Figure 9 is a flowchart showing an example of the process in step S2. [Figure 11] This figure shows an example of the hardware configuration of an information processing device. [Modes for carrying out the invention]

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings. In the description of the drawings, the same or equivalent elements will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] Figure 1 shows an example of the functional configuration of an information processing device 10 according to one embodiment. For example, the information processing device 10 performs information processing to acquire data on the actions (e.g., movement history (movement routes), etc.) of a specific object (e.g., a person such as a customer) within a certain facility (e.g., a commercial facility with a relatively large site such as a department store). The information processing device 10 (analysis unit 12, described later) also has a function to identify objects contained in an image by performing image processing on an image that includes a specific object having multiple feature points as its subject. For example, a specific object having multiple feature points is a person. However, the specific object to be identified by the information processing device 10 is not limited to a person, but may be a living organism other than a person, such as a dog or cat, or a non-living thing such as a robot.

[0012] As shown in Figure 1, the information processing device 10 includes an acquisition unit 11, an analysis unit 12, a generation unit 13, an association information storage unit 14, a person information storage unit 15, and a movement history storage unit 16.

[0013] The acquisition unit 11 acquires association information which associates images taken by each of a plurality of cameras C having different shooting ranges, time information indicating the time when the images were taken, and camera information for identifying the camera C that took the images. The plurality of cameras C are, for example, fixed cameras placed in different locations within a facility. The plurality of cameras C may be installed in various locations within a commercial facility such as a department store, such as the entrance, elevator landings on each floor, corridors on each floor, and individual stores. The camera information serves as information for identifying which camera C (i.e., which camera C installed at which location) took the image. In other words, based on the camera information included in the association information, it is possible to identify the location where the image included in the association information was taken (i.e., the shooting range of the camera C identified by the camera information).

[0014] For example, each of the multiple cameras C is configured to communicate with the information processing device 10 via a wired or wireless communication network. In this case, the acquisition unit 11 can acquire images by receiving images captured from each of the multiple cameras C. The images may be still images or moving images. In the latter case, the acquisition unit 11 sequentially acquires images corresponding to each frame that makes up the moving image. In this embodiment, as an example, the acquisition unit 11 is configured to acquire still images from each camera C periodically (for example, at predetermined intervals of a few seconds).

[0015] As an example, the data transmitted from each camera C to the acquisition unit 11 includes image information indicating the image content (such as the pixel value of each pixel), as well as the time when the image was captured (time information) and camera information for identifying the camera C. The acquisition unit 11 associates a unique image ID (information for identifying an image) with these pieces of information (image information, time information, and camera information) acquired from the camera C. As a result, association information in which the image ID, time information, camera information, and image information are associated with one another is obtained. The association information thus obtained by the acquisition unit 11 is stored (accumulated) in the association information storage unit 14.

[0016] Figure 2 is a diagram illustrating an example of the association information. In this example, the image ID "IM001" is associated with the image captured at "15:00 on March 1, 2023" by the camera C identified by the camera information "C001", so that one piece (for one record) of association information is generated. Similarly, the image ID "IM102" is associated with the image captured at "15:10 on March 1, 2023" by the camera C identified by the camera information "C002", so that one piece (for one record) of association information is generated. As described above, in the present embodiment, as an example, every time an image captured by a certain camera C is transmitted to the information processing apparatus 10, one piece of association information is generated by the acquisition unit 11 and stored in the association information storage unit 14.

[0017] The analysis unit 12 analyzes the images included in the association information acquired by the acquisition unit 11, so as to identify a plurality of images that include a person (specific object) estimated to be the same as a common subject.

[0018] Figure 3 shows an example of the functional configuration of the analysis unit 12. As shown in Figure 3, as an example, the analysis unit 12 includes a feature point detection unit 121, a color information acquisition unit 122, a posture determination unit 123, a person information acquisition unit 124 (object information acquisition unit), and a determination unit 125. The processing of the feature point detection unit 121, the color information acquisition unit 122, the posture determination unit 123, and the person information acquisition unit 124 is performed for each image to be analyzed (i.e., the image included in each associated information).

[0019] The feature point detection unit 121 detects multiple feature points of a person contained in the image to be analyzed (i.e., each image included in each association information). Feature points are characteristic parts that constitute a part of an object (in this embodiment, a person). For example, feature points are parts detected in human skeleton estimation, such as joints like the shoulder, elbow, wrist, and ankle. As shown in Figure 4, in this embodiment, as an example, 11 feature points FP1 to FP11 are used. Feature point FP1 is a feature point corresponding to the center of the torso. Feature point FP2 is a feature point corresponding to the right shoulder. Feature point FP3 is a feature point corresponding to the right wrist. Feature point FP4 is a feature point corresponding to the left shoulder. Feature point FP5 is a feature point corresponding to the left wrist. Feature point FP6 is a feature point corresponding to the hip joint of the right leg. Feature point FP7 is a feature point corresponding to the right knee. Feature point FP8 is a feature point corresponding to the right ankle. Feature point FP9 is a feature point corresponding to the hip joint of the left leg. Feature point FP10 corresponds to the left knee. Feature point FP11 corresponds to the left ankle.

[0020] For example, the feature point detection unit 121 performs a known skeletal position estimation process on the image to be analyzed. The skeletal position estimation process can be performed, for example, by using a machine learning model. Such a machine learning model (trained model) can be obtained, for example, by performing machine learning such as deep learning using training data consisting of training images (images containing people) and corresponding skeletal positions (positions of each feature point) (ground truth labels). Examples of such machine learning models include models using MMPose, OpenPose, etc.

[0021] As an example, as shown in Figure 4(A), if a male person P1 facing forward is included as a subject in the image, the above skeletal position estimation process will yield skeletal information SI1, which includes the positional information of each of the multiple feature points FP1 to FP11 of the person P1. Similarly, as shown in Figure 4(B), if a female person P2 facing forward is included as a subject in the image, the above skeletal position estimation process will yield skeletal information SI2, which includes the positional information of each of the multiple feature points FP1 to FP11 of the person P2.

[0022] The color information acquisition unit 122 acquires color information for each feature point based on the pixel values ​​of the pixels corresponding to each feature point FP1 to FP11 detected by the feature point detection unit 121. As an example, the color information is a color histogram of a region that includes the position (center coordinates) where the feature point was detected and also includes multiple pixels (for example, a circular region with a predetermined radius centered on the center coordinates, as shown in Figure 4).

[0023] Figure 5 is a visual representation of an example of color information (color histogram) corresponding to a single feature point. A color histogram is information that shows the correspondence between each pixel value (0 to 255) and the frequency of occurrence of pixels with each pixel value in an image. Here, the frequency of a certain pixel value is, for example, the number of pixels with that pixel value among the multiple pixels included in the region corresponding to the feature point mentioned above. As shown in Figure 5, as an example, a color histogram is information that shows the above correspondence for each of the R (red), G (green), and B (blue) channels that make up a color image. In this case, the color histogram corresponding to a single feature point can be represented as a "3 (number of channels) × 256 (number of pixel values)" dimensional vector. According to such a color histogram, the hue of the region corresponding to a single feature point can be grasped. In other words, the above color histogram can be said to be information that shows the hue of the region corresponding to a single feature point.

[0024] As an example, the person P1 shown in Figure 4(A) is wearing a white shirt, a gray outer garment, brown trousers, and black shoes. In this case, the above processing by the color information acquisition unit 122 acquires the following color information (color histogram) for each feature point FP1 to FP11 of the skeletal information SI1 corresponding to person P1. Specifically, for feature point FP1, which overlaps with the shirt, a color histogram corresponding to the shirt's color (white) is obtained. For feature points FP2 to FP5, which overlap with the outer garment, a color histogram corresponding to the outer garment's color (gray) is obtained. For feature points FP6, FP7, FP9, and FP10, which overlap with the trousers, a color histogram corresponding to the trousers' color (brown) is obtained. For feature points FP8 and FP11, which overlap with the ankle (shoe), a color histogram corresponding to the shoe's color (black) is obtained.

[0025] As an example, Person P2 shown in Figure 4(B) is wearing a pink shirt, a white outer garment, and brown trousers. Person P2's ankles are exposed. In this case, the above processing by the color information acquisition unit 122 acquires the following color information (color histogram) for each feature point FP1 to FP11 of the skeletal information SI2 corresponding to Person P2. Specifically, for feature point FP1, which overlaps with the shirt, a color histogram corresponding to the shirt's color (pink) is obtained. For feature points FP2 to FP5, which overlap with the outer garment, a color histogram corresponding to the outer garment's color (white) is obtained. For feature points FP6, FP7, FP9, and FP10, which overlap with the trousers, a color histogram corresponding to the trousers' color (brown) is obtained. For feature points FP8 and FP11, which overlap with Person P2's exposed ankles, a color histogram corresponding to the ankle's color (skin tone) is obtained.

[0026] Furthermore, for example, when each of multiple frames (images) constituting a video of a predetermined period (e.g., several seconds) captured by a single camera C is acquired by the acquisition unit 11, even if the position and posture of the subject (person) hardly change during the predetermined period, the color information (color histogram) corresponding to each feature point may change between frames included in the predetermined period due to the amount of light, etc. Therefore, in the above case, the color information acquisition unit 122 may acquire the average value of the color histogram corresponding to a single feature point in each image (each frame) included in the predetermined period as the color information (color histogram) corresponding to that single feature point in each image included in the predetermined period. In other words, the color information acquisition unit 122 may acquire a color histogram obtained from only one specific frame as color information, or it may acquire a color histogram obtained by integrating and averaging the results of multiple frames as color information.

[0027] As described above, the color information for each feature point FP1 to FP11 acquired by the color information acquisition unit 122 (in this embodiment, the color histogram of multiple feature points FP1 to FP11) can be represented as an N-dimensional vector. Here, "N = 11 (number of feature points FP1 to FP11) × 3 (number of channels) × 256 (number of pixel values)". In the following explanation, the color histogram for each feature point FP1 to FP11, which is represented as the N-dimensional vector described above, will be referred to as a "feature vector".

[0028] The posture determination unit 123 determines the posture of a person corresponding to multiple feature points FP1 to FP11 based on the positions of each feature point FP1 to FP11 detected by the feature point detection unit 121 (for example, the positional relationship between each feature point FP1 to FP11). As an example, the posture determination unit 123 determines one of four posture patterns—"facing forward," "facing left," "facing backward," and "facing right"—as the posture of a person corresponding to multiple feature points FP1 to FP11.

[0029] "Front-facing posture" is a posture in which a person is facing the front of the camera. In other words, as shown in the example in Figure 4, the front of the person is captured in the image. "Left-facing posture" is a posture in which the right half of the person's body is facing the front of the camera. In other words, the left-facing posture is a state in which the person has turned 90 degrees to the left from the front-facing posture. "Back-facing posture" is a posture in which the person is facing away from the side of the camera. In other words, the back of the person is captured in the image. "Right-facing posture" is a posture in which the left half of the person's body is facing the front of the camera. In other words, the right-facing posture is a state in which the person has turned 90 degrees to the right from the front-facing posture. Note that the posture patterns determined by the posture determination unit 123 may be further subdivided. In other words, the number of posture patterns determined by the posture determination unit 123 may be 5 or more. Alternatively, in order to simplify the processing, the number of posture patterns may be set to 3 or less.

[0030] Referring to Figure 6, an example of posture determination processing by the posture determination unit 123 in this embodiment will be described. Here, a camera for acquiring images is installed above the space in which a person walks, and the imaging direction of the camera is set to be inclined downward with respect to the horizontal plane. Note that the processing described below using Figure 6 is just one example, and the posture determination unit 123 may perform posture determination processing other than the processing described below depending on the posture pattern used, the position of the camera, etc.

[0031] As shown in (A) of FIG. 6, when the feature point FP2 corresponding to the right shoulder is located on the left side of the feature point FP4 corresponding to the left shoulder (in other words, when the vector from the feature point FP2 toward the feature point FP4 points rightward), the posture determination unit 123 may determine that the posture of the person having these feature points FP2 and FP4 is a "front-facing posture". The above determination processing can be executed, for example, as follows. For example, a vector pointing rightward in the image is set as a reference vector. At this time, the posture determination unit 123 may determine that the posture of the person is the "front-facing posture" when the angle formed by the vector from the feature point FP2 toward the feature point FP4 with respect to the reference vector (the angle with the counterclockwise direction relative to the reference vector defined as the positive direction) is in the vicinity of 0 degrees (for example, when "-t<θ<t" holds for a predetermined threshold t).

[0032] As shown in (B) of FIG. 6, when the feature point FP2 corresponding to the right shoulder is located below the feature point FP4 corresponding to the left shoulder (in other words, when the vector from the feature point FP2 toward the feature point FP4 points upward), the posture determination unit 123 may determine that the posture of the person having these feature points FP2 and FP4 is a "left-facing posture". The above determination processing can be executed, for example, as follows. For example, the posture determination unit 123 may determine that the posture of the person is the "left-facing posture" when the angle formed by the vector from the feature point FP2 toward the feature point FP4 with respect to the reference vector (the angle with the counterclockwise direction relative to the reference vector defined as the positive direction) is in the vicinity of 90 degrees (for example, when "90°-t<θ<90°+t" holds for a predetermined threshold t).

[0033] As shown in Figure 6(C), if the feature point FP2 corresponding to the right shoulder is located to the right of the feature point FP4 corresponding to the left shoulder (in other words, if the vector from feature point FP2 to feature point FP4 is pointing to the left), the posture determination unit 123 may determine that the posture of the person having these feature points FP2 and FP4 is a "rear posture". The above determination process can be performed, for example, as follows. For example, the posture determination unit 123 may determine that the posture of the person is a "rear posture" if the angle that the vector from feature point FP2 to feature point FP4 makes with respect to the reference vector (an angle where counterclockwise is the positive direction with respect to the reference vector) is near 180 degrees (for example, if "180°-t<θ<180°+t" holds for a predetermined threshold t).

[0034] As shown in Figure 6(D), if the feature point FP2 corresponding to the right shoulder is located above the feature point FP4 corresponding to the left shoulder (in other words, if the vector from feature point FP2 to feature point FP4 is downward), the posture determination unit 123 may determine that the posture of the person having these feature points FP2 and FP4 is a "rightward-facing posture". The above determination process can be performed, for example, as follows. For example, the posture determination unit 123 may determine that the posture of the person is a "rightward-facing posture" if the angle that the vector from feature point FP2 to feature point FP4 makes with respect to the reference vector (an angle where counterclockwise rotation with respect to the reference vector is considered positive) is near 270 degrees (for example, if "270°-t<θ<270°+t" holds true for a predetermined threshold t).

[0035] The person information acquisition unit 124 acquires (generates) person information for each image, relating the person, the person's posture, and the color information for each feature point. As shown in Figure 7, in this embodiment, the person information is information relating the image ID, the person ID that identifies the person included as a subject in the image corresponding to the image ID (in this embodiment, skeletal information corresponding to one person detected by the feature point detection unit 121), the posture (in this embodiment, the posture pattern determined by the posture determination unit 123), and the feature vector acquired by the color information acquisition unit 122.

[0036] At the time when person information is acquired by the person information acquisition unit 124, it is unknown whether the person corresponding to that person information is the same person as the person corresponding to already acquired person information. For this reason, the person information acquisition unit 124 assigns a unique person ID to the newly acquired person information that has not been previously assigned. The person information acquired by the person information acquisition unit 124 is stored in the person information storage unit 15. If multiple people are included in the same image, the person information acquisition unit 124 may generate multiple person information corresponding to each of the multiple people, associating them with the same image ID.

[0037] The determination unit 125 determines whether the subject is the same between multiple pieces of association information (i.e., between images corresponding to each piece of association information). Specifically, the determination unit 125 compares the first person information (for example, the person information corresponding to the person ID "P001" in the first row of Figure 7) obtained from the first image (for example, the image corresponding to the image ID "IM001" in Figure 2) which is one of the multiple pieces of association information stored in the association information storage unit 14, with the second person information (for example, the person information corresponding to the person ID "P002" in the second row of Figure 7) obtained from the second image (for example, the image corresponding to the image ID "IM102" in Figure 2) which is another piece of association information included in the multiple pieces of association information. Based on the result of this comparison, the determination unit 125 determines whether the person associated with the first person information (i.e., the person corresponding to person ID "P001") and the person associated with the second person information (i.e., the person corresponding to person ID "P002") are the same person. The determination unit 125 may, for example, perform the above determination process for each of all combinations obtained by extracting any two pieces of association information (the first association information and the second association information) from among a plurality of association pieces of information.

[0038] The determination unit 125 determines, for example, that the person corresponding to the first person information (in this case, the person corresponding to person ID "P001") and the person corresponding to the second person information (in this case, the person corresponding to person ID "P002") are the same person if the posture information contained in the first person information matches the posture information contained in the second person information, and the similarity between the feature vector contained in the first person information and the feature vector contained in the second person information is equal to or greater than a predetermined threshold value. As an example, the cosine distance between the two feature vectors to be compared may be used as the similarity. In this case, the determination unit 125 can determine that the person corresponding to the first person information and the person corresponding to the second person information are the same person if the cosine distance is less than or equal to a predetermined threshold value.

[0039] As described above, if the determination unit 125 determines that person ID "P001" and person ID "P002" are the same person, it may merge multiple person IDs for the same person into a single person ID by changing one person ID to the other person ID. Alternatively, without changing the person IDs that have already been assigned (i.e., the person IDs included in each person information), it may notify the generation unit 13, which will be described later, of information indicating that person ID "P001" and person ID "P002" refer to the same person. In other words, as long as the generation unit 13 can grasp the result of the determination when the processing of the determination unit 125 determines that the person corresponding to one person ID and the person corresponding to another person ID are the same person, the data processing method for this purpose is not particularly limited.

[0040] As a result of the processing by the analysis unit 12 described above, the person IDs of people who are presumed to be the same are merged for the multiple person information individually extracted from the images contained in each of the multiple association pieces of information. As a result, multiple images that contain the same person as a common subject are identified. In other words, in the example described above, as a result of determining that the person corresponding to person ID "P001" and the person corresponding to person ID "P002" are the same, it is identified that the image containing the subject corresponding to person ID "P001" (i.e., the image corresponding to image ID "IM001") and the image containing the subject corresponding to person ID "P002" (i.e., the image corresponding to image ID "IM102") both contain a certain person (i.e., the person corresponding to person ID "P001") as a common subject.

[0041] Furthermore, the feature point detection unit 121, color information acquisition unit 122, posture determination unit 123, and person information acquisition unit 124 may perform processing on each of multiple images in which it is known that the same person is included as the subject (the processing of the feature point detection unit 121, color information acquisition unit 122, posture determination unit 123, and person information acquisition unit 124 as described above) to acquire human body information including feature vectors (color information for each feature point) corresponding to each of multiple different postures for the same person (i.e., the person corresponding to the same person ID). In other words, if it is determined that the subjects included in multiple images are the same person as described above, the person information acquisition unit 124 may assign the same person ID to the person (skeletal information) extracted from each of these multiple images. In addition, the determination unit 125 may use the feature vectors corresponding to each of the multiple different postures to determine whether the person associated with the first person information and the person associated with the second person information are the same person.

[0042] For example, when multiple frames (images) constituting a video of a certain period (e.g., several seconds) captured by a single camera C are acquired by the acquisition unit 11, the same subject (person) included in each frame of the video can be identified and tracked by using known object recognition and object tracking methods. In this case, the posture of the subject in the video may change midway through. That is, the posture of the subject may differ from frame to frame. For example, the posture of the subject (for example, the person corresponding to person ID "P001") may be determined to be "facing forward" in the first frame, and the posture of the subject may be determined to be "facing left" in the second frame. In such a case, the person information acquisition unit 124 may acquire person information for the same person ID "P001" by associating multiple postures (in this example, "facing forward" and "facing left") with feature vectors corresponding to each posture. For example, the person information acquisition unit 124 may acquire person information associated with person ID "P001" and posture "front-facing posture," as well as person information associated with person ID "P001" and posture "left-facing posture." With the above configuration, color information (feature vectors) corresponding to each of multiple postures can be registered for the same person. Then, by performing identity determination by the determination unit 125 using the feature vectors corresponding to the multiple postures registered in this way, the accuracy of identity determination can be improved.

[0043] For example, consider the case where person information (feature vectors) corresponding to the front and back poses of person P3 and person P4, who are different from person P3, as shown in Figure 8, are obtained. Here, the appearance of person P3 in the front pose (the color of each feature point FP1 to FP11) is very similar to the appearance of person P4 in the front pose. That is, the feature vector corresponding to the front pose of person P3 is very similar to the feature vector corresponding to the front pose of person P4. For this reason, if matching (comparison processing) between person P3 and person P4 is performed using only the person information for the front pose, there is a risk of misjudging person P3 and person P4 as the same person. On the other hand, the back of person P4's outer garment has a white cat face design, whereas the back of person P3's outer garment does not have such a design, and only purple fabric is exposed. For this reason, if matching (comparison processing) between person P3 and person P4 is performed using the person information for the back pose as well, it becomes possible to obtain the correct judgment that person P3 and person P4 are not the same person. As described above, by using feature vectors corresponding to each of multiple different poses, the accuracy of matching (accuracy of identity determination) can be improved.

[0044] As described above, the generation unit 13 generates movement history information showing the change in the position of a person (i.e., the person corresponding to person ID "P001") over time (i.e., in this embodiment, the movement path showing how the person moved within the facility) based on the time information and camera information associated with each of the multiple images identified by the analysis unit 12 as containing the same person as the subject (in the above example, the images corresponding to image IDs "IM001" and "IM102"). Here, as an example, suppose that the camera corresponding to camera information "C001" is installed at the entrance of a certain facility (for example, a commercial facility such as a department store), and the camera corresponding to camera information "C002" is installed in the women's clothing section on the 3rd floor of the same facility. In this case, in the above example, movement history information can be generated showing that the person corresponding to person ID "P001" visited the department store at 15:00 on March 1, 2023, and went to the women's clothing section on the 3rd floor of the department store 10 minutes later.

[0045] Furthermore, the data format for movement history information is not limited to a specific format. For example, in the above example, information that lists locations (locations corresponding to camera information) in chronological order, such as "Stayed in the entrance at 3:00 PM on March 1, 2023. Stayed in the women's clothing section on the 3rd floor at 3:10 PM on March 1, 2023," can be used as movement history information. Also, movement history information does not have to be human-readable string information as described above. For example, movement history information may be expressed in a format that is easy for computers to process, such as "202303011500_entrance,202303011510_3Ffujin".

[0046] The movement history information generated by the generation unit 13 is stored in the movement history storage unit 16. The movement history information stored in the movement history storage unit 16 is used, for example, to understand customer movement trends within a facility or to assist in the facility's marketing activities. Furthermore, by analyzing each of the multiple movement history information stored in the movement history storage unit 16, it becomes possible to understand how many customers were gathered at each location at a given time. In other words, since one movement history information is associated with each individual, it becomes possible to understand each person's movement trends and to determine the approximate number of people (unique users) at a particular location at a given time.

[0047] Next, an example of the operation of the information processing device 10 will be described with reference to Figure 9.

[0048] In step S1, the acquisition unit 11 acquires multiple pieces of association information by continuously (for example, periodically) acquiring images (and the time of acquisition, and camera information to identify camera C) taken from each of the multiple cameras C. The association information acquired by the acquisition unit 11 is stored in the association information storage unit 14.

[0049] In step S2, the analysis unit 12 analyzes the images included in the association information acquired by the acquisition unit 11 to identify multiple images that contain the same person as a common subject. In the example described above, the analysis unit 12 determines that the person corresponding to person ID "P001" and the person corresponding to person ID "P002" are the same person, and thereby identifies multiple images (images corresponding to image IDs "IM001" and "IM102") that include these people as subjects.

[0050] Referring to Figure 10, an example of the process in step S2 will be described in detail.

[0051] In step S21, the feature point detection unit 121 detects multiple feature points FP1 to FP11 (see Figure 4) contained in the image to be analyzed (each image included in each association information).

[0052] In step S22, the color information acquisition unit 122 acquires color information for each feature point based on the pixel values ​​of the pixels corresponding to each feature point FP1 to FP11 detected by the feature point detection unit 121 in step S21. In this embodiment, as an example, the color information for each feature point is information showing the color histogram (feature vector) for each feature point FP1 to FP11.

[0053] In step S23, the posture determination unit 123 determines the posture of the person P corresponding to the multiple feature points FP1 to FP11 based on the positions of each feature point FP1 to FP11 detected in step S21. In this embodiment, as an example, the posture determination unit 123 determines the posture based on the positional relationship between feature point FP2 and feature point FP4, as shown in Figure 6.

[0054] In step S24, the person information acquisition unit 124 acquires (generates) person information (see Figure 7) that associates the processing results (color information and posture) from steps S22 and S23 with the image ID and person ID, and stores it in the person information storage unit 15.

[0055] In step S25, the determination unit 125 compares the first person information obtained from the first image included in the first association information (for example, the image corresponding to the image ID "IM001" in Figure 2) (for example, the person information corresponding to the person ID "P001" in the first row of Figure 7) with the second person information obtained from the second image included in the second association information (for example, the image corresponding to the image ID "IM102" in Figure 2) (for example, the person information corresponding to the person ID "P002" in the second row of Figure 7).

[0056] In step S26, the determination unit 125 determines whether the person corresponding to the first person information (here, the person corresponding to person ID "P001") and the person corresponding to the second person information (here, the person corresponding to person ID "P002") are the same person. For example, if the posture information included in the first person information matches the posture information included in the second person information, and the similarity between the feature vector included in the first person information and the feature vector included in the second person information is greater than or equal to a predetermined threshold value, it is determined that the person corresponding to the first person information and the person corresponding to the second person information are the same person (step S26: YES). On the other hand, if the posture information included in the first person information does not match the posture information included in the second person information, or if the similarity between the feature vector included in the first person information and the feature vector included in the second person information is less than the threshold value, it is determined that the person corresponding to the first person information and the person corresponding to the second person information are not the same person (step S26: NO).

[0057] If it is determined that the person corresponding to the first person information and the person corresponding to the second person information are the same person (step S26: YES), the determination unit 125 merges the person ID corresponding to the first person information and the person ID corresponding to the second person information into a common person ID (step S27). As a result, the generation unit 13, which will be described later, will be able to recognize that the person corresponding to the first person information and the person corresponding to the second person information are the same person.

[0058] Returning to Figure 9, in step S3, the generation unit 13 generates movement history information that shows the time change in the position of a specific person (i.e., the person corresponding to person ID "P001") that is included as a common subject in the multiple images (i.e., the movement path that shows how the person moved within the facility) based on the time information and camera information associated with each of the multiple images identified by the analysis unit 12 in step S2 (in the above example, the images corresponding to image IDs "IM001" and "IM102"), and stores it in the movement history storage unit 16.

[0059] In the information processing device 10 described above, multiple images containing a common subject (a specific person) are extracted by analyzing each image captured by each of the multiple cameras C. Then, based on the time information and camera information associated with each of these multiple images, movement history information (i.e., the change in the specific person's position over time) is generated, indicating at what time and where (i.e., the location corresponding to the shooting range of camera C identified by the camera information) the specific person was. Therefore, with the above configuration, data (i.e., movement history information) regarding the actions of a specific object (in this embodiment, a customer visiting a certain facility) can be suitably obtained.

[0060] Furthermore, the information processing device 10 includes an analysis unit 12 having a functional configuration as illustrated in Figure 3. With this configuration, by using the color information of each feature point FP1 to FP11 as a comparison target, the identity of a person can be determined by relatively simple processing. Moreover, by considering the posture of the person corresponding to multiple feature points FP1 to FP11, that is, the color information of each feature point FP1 to FP11 as described above, from which direction the person is viewed, the accuracy of identity determination based on color information can be improved. Therefore, the information processing device 10 can accurately determine the identity of a person by relatively simple processing.

[0061] Furthermore, in this embodiment, the analysis unit 12 identifies multiple images containing the same person as the subject without using facial information of the person. In other words, the information processing device 10 does not perform processes such as facial recognition using facial information to determine the identity of the person. Thus, since the information processing device 10 does not need to use privacy information such as a person's face to identify a person, it becomes easier to protect the privacy of a person who appears as a subject in an image. For example, even if image processing is performed to automatically detect the face portion of a person that appears in an image and replace it with another image in order to protect privacy, the identity of the person can be appropriately determined by using feature points FP1 to FP11 other than the face, as described in the above embodiment.

[0062] In the above embodiment, the analysis unit 12 is configured to use a combination of color information and pose for each feature point as a method for determining identity. However, the analysis unit 12 may identify multiple images that contain the same person as a common subject by using other methods for determining identity besides the above method. Examples of other methods for determining identity include, for example, methods for estimating the size (height, etc.) of a subject using known depth estimation techniques, and methods for estimating the three-dimensional position of a subject. By using such methods, identity determination can be performed without using facial information of the person. Furthermore, the analysis unit 12 may use multiple identity determination methods as described above. For example, if the same result is obtained in each of the multiple identity determination methods, the analysis unit 12 may adopt that result. The above configuration makes it possible to improve the accuracy of identity determination.

[0063] The information processing device 10 of this disclosure has the following configuration.

[0064] [1] An acquisition unit that acquires association information relating images taken by each of a plurality of cameras having different shooting ranges, time information indicating the time when the images were taken, and camera information for identifying the camera that took the images, An analysis unit identifies multiple images that include a specific object presumed to be the same as a common subject by analyzing the images included in the association information acquired by the acquisition unit, A generation unit generates movement history information indicating the time change of the position of the specific object based on the time information and camera information associated with each of the plurality of images identified by the analysis unit, An information processing device equipped with the following features.

[0065] [2] The analysis unit is, A feature point detection unit that detects multiple feature points of an object included in the image to be analyzed, A color information acquisition unit acquires color information for each feature point based on the pixel value of the pixel corresponding to each detected feature point, An attitude determination unit that determines the attitude of the object based on the position of each of the detected feature points, An object information acquisition unit acquires object information for each image, relating the object, the object's orientation, and the color information for each feature point to each other. The system includes a determination unit that determines whether the object associated with the first object information and the object associated with the second object information are the same, based on a comparison result between the first object information obtained from the first image included in the first association information and the second object information obtained from the second image included in the second association information. [1] Information processing device.

[0066] [3] The feature point detection unit, the color information acquisition unit, the posture determination unit, and the object information acquisition unit perform processing on each of a plurality of images in which it is known that the same object is included as a subject, thereby acquiring object information for the same object, including color information for each of the feature points corresponding to each of a plurality of different postures, The determination unit determines whether the object associated with the first object information and the object associated with the second object information are the same, by using the color information for each of the feature points corresponding to each of the plurality of different poses. [2] Information processing device.

[0067] [4] The object is a person, The analysis unit identifies the plurality of images that include the same person as the subject, without using facial information of the person. [1] to [3] one of the following information processing devices

[0068] Furthermore, the block diagrams used in the description of the above embodiments show functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Moreover, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may be realized by combining the above one device or the above multiple devices with software.

[0069] Functions include, but are not limited to, judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, deem, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning.

[0070] For example, the information processing device 10 in one embodiment of the present disclosure may function as a computer that performs the object identification method of the present disclosure. Figure 11 is a diagram showing an example of the hardware configuration of the information processing device 10 according to one embodiment of the present disclosure. Physically, the information processing device 10 may be configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, etc. The user terminal 20 may also have a hardware configuration similar to that of the information processing device 10.

[0071] In the following explanation, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware configuration of the information processing device 10 may include one or more of the devices shown in Figure 11, or it may be configured to omit some of the devices.

[0072] Each function in the information processing device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, which allows the processor 1001 to perform calculations, control communication by the communication device 1004, and control at least one of data reading and writing in the memory 1002 and storage 1003.

[0073] The processor 1001 controls the entire computer, for example, by running an operating system. The processor 1001 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, etc.

[0074] Furthermore, the processor 1001 reads programs (program code), software modules, data, etc., from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described in the above embodiment. For example, each functional unit of the information processing device 10 (e.g., the analysis unit 12) may be stored in the memory 1002 and implemented by a control program that runs on the processor 1001, and other functional blocks may be implemented similarly. The above-described various processes have been explained as being executed by one processor 1001, but they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from a network via a telecommunications line.

[0075] Memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be called a register, cache, main memory, etc. Memory 1002 can store executable programs (program code), software modules, etc., for carrying out an information processing method according to one embodiment of the present disclosure.

[0076] Storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. Storage 1003 may also be called an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, server, or other suitable medium including at least one of memory 1002 and storage 1003.

[0077] The communication device 1004 is hardware (transceiver / receiver device) for communicating between computers via at least one of a wired network and a wireless network, and is also called a network device, network controller, network card, communication module, etc.

[0078] The input device 1005 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.). The output device 1006 is an output device that outputs to an external source (e.g., a display, speaker, LED lamp, etc.). The input device 1005 and the output device 1006 may be configured as an integrated unit (e.g., a touch panel).

[0079] Furthermore, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or different buses may be configured for each device.

[0080] Furthermore, the information processing device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.

[0081] Although this embodiment has been described in detail above, it will be clear to those skilled in the art that this embodiment is not limited to the embodiments described herein. This embodiment can be implemented as a modified and altered form without departing from the spirit and scope of the invention as defined by the claims. Therefore, the description herein is for illustrative purposes only and is not intended to be restrictive in any way to this embodiment.

[0082] The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described herein may be reordered, provided they are consistent with each other. For example, the methods described herein present various step elements in an exemplary order and are not limited to that specific order.

[0083] Input and output information may be stored in a specific location (e.g., memory) or managed using a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be transmitted to other devices.

[0084] The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0085] Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of specific information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0086] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, and so on, whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name.

[0087] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0088] The information, signals, etc. described in this disclosure may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0089] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0090] The names used for the parameters described above are not restrictive in any way. Furthermore, the formulas and other expressions using these parameters may differ from those expressly disclosed in this disclosure. Since various information elements can be identified by any suitable name, the various names assigned to these various information elements are not restrictive in any way.

[0091] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0092] Any reference to elements using the designations “first,” “second,” etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements do not imply that only two elements may be employed, or that the first element must precede the second element in any way.

[0093] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0094] In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0095] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different." [Explanation of symbols]

[0096] 10... Information processing device, 11... Acquisition unit, 12... Analysis unit, 13... Generation unit, 121... Feature point detection unit, 122... Color information acquisition unit, 123... Pose determination unit, 124... Person information acquisition unit (object information acquisition unit), 125... Determination unit, C... Camera, FP1~FP11... Feature points.

Claims

1. An acquisition unit acquires association information which associates images taken by each of a plurality of cameras having different shooting ranges, time information indicating the time when the images were taken, and camera information for identifying the camera that took the images. An analysis unit identifies multiple images that include a specific object presumed to be the same as a common subject by analyzing the images included in the association information acquired by the acquisition unit, A generation unit generates movement history information indicating the time change of the position of the specific object based on the time information and camera information associated with each of the plurality of images identified by the analysis unit, Equipped with, The aforementioned analysis unit, A feature point detection unit that detects multiple feature points of an object included in the image to be analyzed, A color information acquisition unit acquires color information for each feature point based on the pixel value of the pixel corresponding to each detected feature point, An attitude determination unit that determines the attitude of the object based on the position of each of the detected feature points, An object information acquisition unit acquires object information for each image, relating the object, the object's orientation, and the color information for each feature point to each other. The system includes a determination unit that determines whether the object associated with the first object information and the object associated with the second object information are the same, based on the comparison result of the first object information and the second object information. The feature point detection unit, the color information acquisition unit, the posture determination unit, and the object information acquisition unit perform processing on each of a plurality of images in which the same object is known to be included as the subject, thereby acquiring the first object information and the second object information, respectively, as object information including color information for each of the feature points corresponding to each of a plurality of different postures of the same object. The determination unit determines whether the object associated with the first object information and the object associated with the second object information are the same, based on the results of both comparisons: a first comparison between the color information for each feature point corresponding to the first pose included in the first object information and the color information for each feature point corresponding to the first pose included in the second object information and the color information for each feature point corresponding to the second pose included in the second object information. Information processing device.

2. The aforementioned object is a person, The analysis unit identifies the plurality of images that include the same person as the subject, without using facial information of the person. The information processing apparatus according to claim 1.

Citation Information

Patent Citations

  • Imaging device and program thereof

    JP2008028747A

  • User detecting apparatus, user detecting method, and user detecting program

    JP2013125340A

  • Determination device, determination method, and determination program

    JP2016009448A

  • Image management device and image management method

    JP2016048515A

  • Image determination device, image determination method, and program

    JP2016162099A